#20612scandinavian letters are transliterated wrong
Description
The scandinavian letters ä, ö and å are rendered, with for example realurl, in a wrong way. Ä is ae, ö is oe.. Ä should be a, ö o and å a.
This should be added to the file typo3_src-X/t3lib/unidata/Translit.txt:
- scandinavian
00e4; 0061; LATIN SMALL LETTER A WITH UMLAUTS => a (finnish)
00c4; 0041; LATIN CAPITAL LETTER A WITH UMLAUTS => A (finnish)
00f6; 006f; LATIN SMALL LETTER O WITH UMLAUTS => o (finnish)
00d6; 004f; LATIN CAPITAL LETTER O WITH UMLAUTS => O (finnish)
00e5; 0061; LATIN SMALL LETTER SWEDISH A (Å) => a (finnish)
00c5; 0041; LATIN CAPITAL LETTER SWEDISH A (Å) => a (finnish)
(issue imported from #M11322)
Reviews 3
Related issues 3
Files 4
History 3552
Show 2627 earlier messages
-
Martin Kutschker commented <span data-processor="relativetime" data-value="2009-06-18T14:05:48+00:00" > 06/18/09, 14:05 </span>
-
Alexander Opitz commented and changed status
NewNeeds Feedback target version TYPO3 Version 4.2 <span data-processor="relativetime" data-value="2013-05-08T21:14:54+00:00" > 05/08/13, 21:14 </span>The issue is very old, does this issue exists in newer versions of TYPO3 CMS (4.5 or 6.1)?
-
Katja Lampela commented <span data-processor="relativetime" data-value="2013-05-09T06:12:08+00:00" > 05/09/13, 06:12 </span>
Yes it still exists in 4.5-4.7. Haven’t tried 6, but I suspect nothing has been done there either.
-
Alexander Opitz changed status
Needs FeedbackNew <span data-processor="relativetime" data-value="2013-05-09T08:34:53+00:00" > 05/09/13, 08:34 </span> -
Mathias Schreiber changed target version Is Regression 0 <span data-processor="relativetime" data-value="2015-01-16T17:20:03+00:00" > 01/16/15, 17:20 </span>
-
Riccardo De Contardi commented <span data-processor="relativetime" data-value="2015-04-08T12:03:54+00:00" > 04/08/15, 12:03 </span>
Still present in 6.2.11 I guess, the file /typo3/sysext/core/Resources/Private/Charsets/unidata/Translit.txt does not contain a “Scandinavian” section.
What about adding an option (in Install tool?) where to specify a custom file instead of modifying the original in the core?
-
Benni Mack changed target version <span data-processor="relativetime" data-value="2015-06-15T18:12:02+00:00" > 06/15/15, 18:12 </span>
-
Susanne Moog changed target version <span data-processor="relativetime" data-value="2015-08-05T10:31:17+00:00" > 08/05/15, 10:31 </span>
-
Joonas Kauhanen commented <span data-processor="relativetime" data-value="2015-09-08T05:21:36+00:00" > 09/08/15, 05:21 </span>
Riccardo De Contardi wrote:
> Still present in 6.2.11 I guess, the file /typo3/sysext/core/Resources/Private/Charsets/unidata/Translit.txt does not contain a “Scandinavian” section.
>
> What about adding an option (in Install tool?) where to specify a custom file instead of modifying the original in the core?This would be very handy. Now we have to manually update the Translit.txt file every time after updating TYPO3 source. As many of the developers are from central Europe, they don’t understand the issue. And I think it’s not an issue strictly to scandinavians, as the transliterations should really be language specific, not system wide.
Hoping this to be part of 7 LTS
-
Benni Mack changed target version <span data-processor="relativetime" data-value="2015-09-24T20:42:26+00:00" > 09/24/15, 20:42 </span>
-
Jan Helke changed assigned to Jan Helke <span data-processor="relativetime" data-value="2015-10-16T18:24:23+00:00" > 10/16/15, 18:24 </span>
-
Gerrit Code Review changed status
NewUnder Review <span data-processor="relativetime" data-value="2015-10-16T18:32:25+00:00" > 10/16/15, 18:32 </span> Patch set 1 for branch *master* of project *Packages/TYPO3.CMS* has been pushed to the review server. It is available at http://review.typo3.org/44115 -
Christian Kuhn changed status
Under ReviewNew <span data-processor="relativetime" data-value="2015-10-29T09:52:15+00:00" > 10/29/15, 09:52 </span> -
Riccardo De Contardi changed target version <span data-processor="relativetime" data-value="2016-03-18T15:24:28+00:00" > 03/18/16, 15:24 </span>
-
Bart Lammers commented <span data-processor="relativetime" data-value="2016-10-26T10:12:19+00:00" > 10/26/16, 10:12 </span>
This still seems to be the case. For a client this is a big issue in regards to speaking URLs that have (in their eyes) wrong letters in them.
The solution as provided earlier by Jan Helke seems like a nice solution, using the Install Tool to override the default transliteration files.Back then the reasoning to not accept this was to rework the CharsetConverter completely, but now all trough version 4.x, 6.2LTS, 7.6LTS and now 8.x this has not been addressed.
Is it possible to reconsider the given solution?
-
Benni Mack commented <span data-processor="relativetime" data-value="2017-05-10T21:41:24+00:00" > 05/10/17, 21:41 </span>
Hey Bart,
maybe we can use a PHP library to do the work now that we cleaned up CharsetConverter big time, do you know any good places to look for?
-
Susanne Moog changed category removed the assignee <span data-processor="relativetime" data-value="2017-10-29T16:45:15+00:00" > 10/29/17, 16:45 </span>
-
Markus Klein changed Duplicated by #83438 <span data-processor="relativetime" data-value="2018-01-11T13:59:23+00:00" > 01/11/18, 13:59 </span>
-
Christian Kuhn changed Related to #83546 <span data-processor="relativetime" data-value="2018-01-12T13:25:23+00:00" > 01/12/18, 13:25 </span>
-
Benni Mack commented and changed status
NewNeeds Feedback <span data-processor="relativetime" data-value="2018-09-30T00:46:38+00:00" > 09/30/18, 00:46 </span>Hey,
this issue should be fixed with 9 LTS and site handling. Please let us know if the new version will solve your issue, otherwise we’ll close this ticket in the next weeks.
Benni.
-
Joonas Kauhanen commented <span data-processor="relativetime" data-value="2018-10-29T16:19:28+00:00" > 10/29/18, 16:19 </span>
We have now tested the new 9 LTS with Site Handling and Url Routing. Unfortunately the issue is still not completely resolved, but we are getting near!
Let me go through the issue by example:
- Editor creates a page in Finnish language, titled “Ääni” (that is “Sound” in Finnish)
- In the Page properties, TYPO3 shows generated URL Segment “/aeaeni“
- This is incorrect for Finnish (and Swedish, Danish, etc) language, so the editor must manually override the URL Segment as “/aani”For pages, manually correcting all URL segments is manageable, but somehow irritating. However, if the URL contains content from other records such as News or an integrated Product database, there may not even be an opportunity to input correct URL segments and all the generated slugs are translittered wrong.
I have tracked the URL generation down to
sysext/core/Classes/DataHandling/SlugHelper.phpwhere thefunction sanitizeis used to make slugs URL compatible. This function usesCharsetConverter->specCharsToASCIIto convert extended letters to ASCII characters. No locale or language parameters are provided, and theCharsetConverteruses a hardcoded path toResources/Private/Charsets/unidata/Translit.txtwhere the character replacements are defined. This file is only compatible with German way of spelling umlaut characters.Do you have ideas how this could be improved so that we could somehow provide conversion tables for languages other than German? Essentially we still have to edit
Resources/Private/Charsets/unidata/Translit.txtmanually after updating TYPO3 and remember to cleartypo3tempcache after that. -
Susanne Moog commented <span data-processor="relativetime" data-value="2019-03-04T22:48:56+00:00" > 03/04/19, 22:48 </span>
Some research notes:
- PHP has a Transliterator in the `intl` extension, but we’d need to provide custom rules there, too.
- Our Symfony `intl` polyfill does not polyfill the Transliterator meaning we’d have to introduce `intl` as hard dependency
- There seems to be no decent PHP transliteration solution covering our use cases out of the boxMay be possible:
- Provide the possibility to register own translit.txt file in localconf to allow integrators to use custom transliterations -
Alexander Schnitzler commented <span data-processor="relativetime" data-value="2019-12-22T16:41:40+00:00" > 12/22/19, 16:41 </span>
Susanne Moog wrote:
> Some research notes:
>
> – PHP has a Transliterator in the `intl` extension, but we’d need to provide custom rules there, too.
> – Our Symfony `intl` polyfill does not polyfill the Transliterator meaning we’d have to introduce `intl` as hard dependency
> – There seems to be no decent PHP transliteration solution covering our use cases out of the box
>
> May be possible:
> – Provide the possibility to register own translit.txt file in localconf to allow integrators to use custom transliterationsJust a short notice that symfony/string, which has been released just a couple of days ago, does a really great job, normalizing and converting string, respecting locales properly:
(new AsciiSlugger('de'))->slug('Näe ja koe')->toString(); // Naee-ja-koe (new AsciiSlugger('dk'))->slug('Näe ja koe')->toString(); // Nae-ja-koeWe should evaluate how much own logic we actually need if using this library.
Could replace our current slugger and parts of the CharsetConverter (maybe even all of it). -
Mathias Bolt Lesniak commented and changed attached transliterate-norwegian.png attached transliterate-english.png attached transliterate-german.png attached transliterate-swedish.png <span data-processor="relativetime" data-value="2020-09-10T14:50:23+00:00" > 09/10/20, 14:50 </span>
Alexander Schnitzler wrote:
> We should evaluate how much own logic we actually need if using this library.
> Could replace our current slugger and parts of the CharsetConverter (maybe even all of it).I made a first implementation the AsciiSlug class in symfony/string. It’s working OK, but I notice TYPO3 doesn’t transliterate e.g. Chinese and Hindi characters. AsciiSlug does.
This implementation uses the page language to determine how to transliterate the string, so & can be transliterated as “og” in Norwegian and “und” in German.
Implementation: https://github.com/pixelant/transliterator
-
Christoph Lehmann changed Related to #93883 <span data-processor="relativetime" data-value="2021-04-11T14:09:17+00:00" > 04/11/21, 14:09 </span>
-
Martin Kutschker commented <span data-processor="relativetime" data-value="2021-04-11T16:14:37+00:00" > 04/11/21, 16:14 </span>
Today IMHO all the transliteration is not needed at all. All the world is happy with Unicode.
-
Joonas Kauhanen commented <span data-processor="relativetime" data-value="2021-04-12T00:44:46+00:00" > 04/12/21, 00:44 </span>
Martin Kutschker wrote in #note-26:
> Today IMHO all the transliteration is not needed at all. All the world is happy with Unicode.That is a great solution, if we can use Unicode in URLs.
Modern browsers seem to support it, but to be sure the URLs should still probably be percent-encoded?
-
Mathias Bolt Lesniak commented <span data-processor="relativetime" data-value="2021-08-13T06:15:06+00:00" > 08/13/21, 06:15 </span>
Just for the sake of completeness, transliteration doesn’t only apply to slugs and URLs in TYPO3. It also applies to file names, where UTF-8 encoding may still not be working.
-
Paul Hansen commented <span data-processor="relativetime" data-value="2023-12-04T23:21:52+00:00" > 12/04/23, 23:21 </span>
After some research, it looks like modern browsers will seamlessly handle percent-encoded URLs for more thorough l10n.
Symfony supported this as of version 6.1 in 2022: https://symfony.com/blog/new-in-symfony-6-1-improved-routing-requirements-and-utf-8-parameters
Just look around Wikipedia and you’ll see this in action. This doc provides some examples: https://developers.google.com/search/docs/crawling-indexing/url-structure
-
Mathias Bolt Lesniak commented <span data-processor="relativetime" data-value="2023-12-05T02:17:40+00:00" > 12/05/23, 02:17 </span>
Paul Hansen wrote in #note-29:
> After some research, it looks like modern browsers will seamlessly handle percent-encoded URLs for more thorough l10n.Yes that is absolutely true. At the same time, it doesn’t remove the need for transliteration — especially where typing is involved. “That accented letter isn’t on my keyboard” or “how do I type 漢?”
Mathias
-
Garvin Hicking changed subject
scandinavian letters are translittered wrongscandinavian letters are transliterated wrong Tags translit,transliteration,transliterate,slugs,utf8,latin <span data-processor="relativetime" data-value="2024-07-01T15:10:13+00:00" > 07/01/24, 15:10 </span> -
Gerrit Code Review pushed patch set 1 for main to change 85574 status
Needs FeedbackUnder Review <span data-processor="relativetime" data-value="2024-08-09T14:34:28+00:00" > 08/09/24, 14:34 </span> -
Gerrit Code Review pushed patch set 2 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-12T09:21:50+00:00" > 08/12/24, 09:21 </span>
-
Gerrit Code Review pushed patch set 3 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-12T09:35:26+00:00" > 08/12/24, 09:35 </span>
-
Gerrit Code Review pushed patch set 4 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-12T12:36:43+00:00" > 08/12/24, 12:36 </span>
-
Gerrit Code Review pushed patch set 5 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-13T08:32:10+00:00" > 08/13/24, 08:32 </span>
-
Gerrit Code Review pushed patch set 6 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-13T10:40:09+00:00" > 08/13/24, 10:40 </span>
-
Gerrit Code Review pushed patch set 7 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-13T13:05:38+00:00" > 08/13/24, 13:05 </span>
-
Gerrit Code Review pushed patch set 8 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-13T13:11:23+00:00" > 08/13/24, 13:11 </span>
-
Gerrit Code Review pushed patch set 9 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-13T13:13:25+00:00" > 08/13/24, 13:13 </span>
-
Gerrit Code Review pushed patch set 10 for main to change 85574 <span data-processor="relativetime" data-value="2024-08-13T13:18:35+00:00" > 08/13/24, 13:18 </span>
-
Gerrit Code Review pushed patch set 11 for main to change 85574 <span data-processor="relativetime" data-value="2024-09-10T11:25:50+00:00" > 09/10/24, 11:25 </span>
-
Gerrit Code Review pushed patch set 12 for main to change 85574 <span data-processor="relativetime" data-value="2024-09-11T23:22:11+00:00" > 09/11/24, 23:22 </span>
-
Gerrit Code Review pushed patch set 13 for main to change 85574 <span data-processor="relativetime" data-value="2024-09-11T23:46:58+00:00" > 09/11/24, 23:46 </span>
-
Gerald Rintisch commented <span data-processor="relativetime" data-value="2025-04-14T03:58:41+00:00" > 04/14/25, 03:58 </span>
Thanks for all the work!
I would really love to see this feature in v14. I am building a website for Sweden and needed to rebase this patch.
It works and is a must-have to use TYPO3 in a Scandinavian country.
Too many companies are using WP or C#-based systems (← not really open-source). To compete this feature is absolutely needed :) -
Gerrit Code Review pushed patch set 14 for main to change 85574 <span data-processor="relativetime" data-value="2025-07-08T07:37:26+00:00" > 07/08/25, 07:37 </span>
-
Gerrit Code Review pushed patch set 15 for main to change 85574 <span data-processor="relativetime" data-value="2025-07-08T07:52:35+00:00" > 07/08/25, 07:52 </span>
-
Gerrit Code Review pushed patch set 16 for main to change 85574 <span data-processor="relativetime" data-value="2025-07-08T07:53:23+00:00" > 07/08/25, 07:53 </span>
-
Gerald Rintisch commented <span data-processor="relativetime" data-value="2026-01-05T07:58:40+00:00" > 01/05/26, 07:58 </span>
Hello!
Are there plans to make this working for TYPO3 v14? -
Garvin Hicking changed Duplicated by #109828 <span data-processor="relativetime" data-value="2026-05-13T15:08:38+00:00" > 05/13/26, 15:08 </span>
-
Benni Mack commented <span data-processor="relativetime" data-value="2026-10-23T08:04:56+00:00" > 10/23/26, 08:04 </span>
Gerald Rintisch wrote in #note-49:
> Hello!
> Are there plans to make this working for TYPO3 v14?I think we need to define hwere we want it – slugs, file names etc?
-
Mathias Bolt Lesniak commented <span data-processor="relativetime" data-value="2026-10-23T08:35:49+00:00" > 10/23/26, 08:35 </span>
Benni Mack wrote in #note-51:
> Gerald Rintisch wrote in #note-49:
> > Hello!
> > Are there plans to make this working for TYPO3 v14?
>
> I think we need to define hwere we want it – slugs, file names etc?This is basically about replacing the transliteration library, so anywhere transliteration happens. Ideally, it should be possible to use different implementations. A lot of progress on this has been made as a part of https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574 . It’s currently blocked by the way tests are set up and the DataHandler.
I’m happy to jump back into this at any time.
Unfortunately the transliteration is currently language independent. The vowels with umlauts are transliterated according to German custom. Which is fine for the huge German user base.
As a workaround you may change Translit.txt and and delete all files in typo3temp/cs.