Issue Bug Under Review Quiet

#20612scandinavian letters are transliterated wrong

Katja Lampela 06/12/09, 22:24 updated 10/23/26, 08:35

Description

The scandinavian letters ä, ö and å are rendered, with for example realurl, in a wrong way. Ä is ae, ö is oe.. Ä should be a, ö o and å a.

This should be added to the file typo3_src-X/t3lib/unidata/Translit.txt:

  1. scandinavian
    00e4; 0061; LATIN SMALL LETTER A WITH UMLAUTS => a (finnish)
    00c4; 0041; LATIN CAPITAL LETTER A WITH UMLAUTS => A (finnish)
    00f6; 006f; LATIN SMALL LETTER O WITH UMLAUTS => o (finnish)
    00d6; 004f; LATIN CAPITAL LETTER O WITH UMLAUTS => O (finnish)
    00e5; 0061; LATIN SMALL LETTER SWEDISH A (Å) => a (finnish)
    00c5; 0041; LATIN CAPITAL LETTER SWEDISH A (Å) => a (finnish)
    (issue imported from #M11322)

Reviews 3

Files 4

History 3552

Show 2627 earlier messages
  1. Martin Kutschker commented <span data-processor="relativetime" data-value="2009-06-18T14:05:48+00:00" > 06/18/09, 14:05 </span>

    Unfortunately the transliteration is currently language independent. The vowels with umlauts are transliterated according to German custom. Which is fine for the huge German user base.

    As a workaround you may change Translit.txt and and delete all files in typo3temp/cs.

  2. Alexander Opitz commented and changed status New Needs Feedback target version TYPO3 Version 4.2 <span data-processor="relativetime" data-value="2013-05-08T21:14:54+00:00" > 05/08/13, 21:14 </span>

    The issue is very old, does this issue exists in newer versions of TYPO3 CMS (4.5 or 6.1)?

  3. Katja Lampela commented <span data-processor="relativetime" data-value="2013-05-09T06:12:08+00:00" > 05/09/13, 06:12 </span>

    Yes it still exists in 4.5-4.7. Haven’t tried 6, but I suspect nothing has been done there either.

  4. Riccardo De Contardi commented <span data-processor="relativetime" data-value="2015-04-08T12:03:54+00:00" > 04/08/15, 12:03 </span>

    Still present in 6.2.11 I guess, the file /typo3/sysext/core/Resources/Private/Charsets/unidata/Translit.txt does not contain a “Scandinavian” section.

    What about adding an option (in Install tool?) where to specify a custom file instead of modifying the original in the core?

  5. Joonas Kauhanen commented <span data-processor="relativetime" data-value="2015-09-08T05:21:36+00:00" > 09/08/15, 05:21 </span>

    Riccardo De Contardi wrote:
    > Still present in 6.2.11 I guess, the file /typo3/sysext/core/Resources/Private/Charsets/unidata/Translit.txt does not contain a “Scandinavian” section.
    >
    > What about adding an option (in Install tool?) where to specify a custom file instead of modifying the original in the core?

    This would be very handy. Now we have to manually update the Translit.txt file every time after updating TYPO3 source. As many of the developers are from central Europe, they don’t understand the issue. And I think it’s not an issue strictly to scandinavians, as the transliterations should really be language specific, not system wide.

    Hoping this to be part of 7 LTS

  6. Gerrit Code Review changed status New Under Review <span data-processor="relativetime" data-value="2015-10-16T18:32:25+00:00" > 10/16/15, 18:32 </span> Patch set 1 for branch *master* of project *Packages/TYPO3.CMS* has been pushed to the review server. It is available at http://review.typo3.org/44115

  7. Bart Lammers commented <span data-processor="relativetime" data-value="2016-10-26T10:12:19+00:00" > 10/26/16, 10:12 </span>

    This still seems to be the case. For a client this is a big issue in regards to speaking URLs that have (in their eyes) wrong letters in them.
    The solution as provided earlier by Jan Helke seems like a nice solution, using the Install Tool to override the default transliteration files.

    Back then the reasoning to not accept this was to rework the CharsetConverter completely, but now all trough version 4.x, 6.2LTS, 7.6LTS and now 8.x this has not been addressed.

    Is it possible to reconsider the given solution?

  8. Benni Mack commented <span data-processor="relativetime" data-value="2017-05-10T21:41:24+00:00" > 05/10/17, 21:41 </span>

    Hey Bart,

    maybe we can use a PHP library to do the work now that we cleaned up CharsetConverter big time, do you know any good places to look for?

  9. Benni Mack commented and changed status New Needs Feedback <span data-processor="relativetime" data-value="2018-09-30T00:46:38+00:00" > 09/30/18, 00:46 </span>

    Hey,

    this issue should be fixed with 9 LTS and site handling. Please let us know if the new version will solve your issue, otherwise we’ll close this ticket in the next weeks.

    Benni.

  10. Joonas Kauhanen commented <span data-processor="relativetime" data-value="2018-10-29T16:19:28+00:00" > 10/29/18, 16:19 </span>

    We have now tested the new 9 LTS with Site Handling and Url Routing. Unfortunately the issue is still not completely resolved, but we are getting near!

    Let me go through the issue by example:

    - Editor creates a page in Finnish language, titled “Ääni” (that is “Sound” in Finnish)
    - In the Page properties, TYPO3 shows generated URL Segment “/aeaeni“
    - This is incorrect for Finnish (and Swedish, Danish, etc) language, so the editor must manually override the URL Segment as “/aani”

    For pages, manually correcting all URL segments is manageable, but somehow irritating. However, if the URL contains content from other records such as News or an integrated Product database, there may not even be an opportunity to input correct URL segments and all the generated slugs are translittered wrong.

    I have tracked the URL generation down to sysext/core/Classes/DataHandling/SlugHelper.php where the function sanitize is used to make slugs URL compatible. This function uses CharsetConverter->specCharsToASCII to convert extended letters to ASCII characters. No locale or language parameters are provided, and the CharsetConverter uses a hardcoded path to Resources/Private/Charsets/unidata/Translit.txt where the character replacements are defined. This file is only compatible with German way of spelling umlaut characters.

    Do you have ideas how this could be improved so that we could somehow provide conversion tables for languages other than German? Essentially we still have to edit Resources/Private/Charsets/unidata/Translit.txt manually after updating TYPO3 and remember to clear typo3temp cache after that.

  11. Susanne Moog commented <span data-processor="relativetime" data-value="2019-03-04T22:48:56+00:00" > 03/04/19, 22:48 </span>

    Some research notes:

    - PHP has a Transliterator in the `intl` extension, but we’d need to provide custom rules there, too.
    - Our Symfony `intl` polyfill does not polyfill the Transliterator meaning we’d have to introduce `intl` as hard dependency
    - There seems to be no decent PHP transliteration solution covering our use cases out of the box

    May be possible:
    - Provide the possibility to register own translit.txt file in localconf to allow integrators to use custom transliterations

  12. Alexander Schnitzler commented <span data-processor="relativetime" data-value="2019-12-22T16:41:40+00:00" > 12/22/19, 16:41 </span>

    Susanne Moog wrote:
    > Some research notes:
    >
    > – PHP has a Transliterator in the `intl` extension, but we’d need to provide custom rules there, too.
    > – Our Symfony `intl` polyfill does not polyfill the Transliterator meaning we’d have to introduce `intl` as hard dependency
    > – There seems to be no decent PHP transliteration solution covering our use cases out of the box
    >
    > May be possible:
    > – Provide the possibility to register own translit.txt file in localconf to allow integrators to use custom transliterations

    Just a short notice that symfony/string, which has been released just a couple of days ago, does a really great job, normalizing and converting string, respecting locales properly:

    (new AsciiSlugger('de'))->slug('Näe ja koe')->toString(); // Naee-ja-koe
    (new AsciiSlugger('dk'))->slug('Näe ja koe')->toString(); // Nae-ja-koe

    We should evaluate how much own logic we actually need if using this library.
    Could replace our current slugger and parts of the CharsetConverter (maybe even all of it).

  13. Mathias Bolt Lesniak commented and changed attached transliterate-norwegian.png attached transliterate-english.png attached transliterate-german.png attached transliterate-swedish.png <span data-processor="relativetime" data-value="2020-09-10T14:50:23+00:00" > 09/10/20, 14:50 </span>

    Alexander Schnitzler wrote:
    > We should evaluate how much own logic we actually need if using this library.
    > Could replace our current slugger and parts of the CharsetConverter (maybe even all of it).

    I made a first implementation the AsciiSlug class in symfony/string. It’s working OK, but I notice TYPO3 doesn’t transliterate e.g. Chinese and Hindi characters. AsciiSlug does.

    This implementation uses the page language to determine how to transliterate the string, so & can be transliterated as “og” in Norwegian and “und” in German.

    Implementation: https://github.com/pixelant/transliterator

  14. Martin Kutschker commented <span data-processor="relativetime" data-value="2021-04-11T16:14:37+00:00" > 04/11/21, 16:14 </span>

    Today IMHO all the transliteration is not needed at all. All the world is happy with Unicode.

  15. Joonas Kauhanen commented <span data-processor="relativetime" data-value="2021-04-12T00:44:46+00:00" > 04/12/21, 00:44 </span>

    Martin Kutschker wrote in #note-26:
    > Today IMHO all the transliteration is not needed at all. All the world is happy with Unicode.

    That is a great solution, if we can use Unicode in URLs.
    Modern browsers seem to support it, but to be sure the URLs should still probably be percent-encoded?

  1. Mathias Bolt Lesniak commented <span data-processor="relativetime" data-value="2021-08-13T06:15:06+00:00" > 08/13/21, 06:15 </span>

    Just for the sake of completeness, transliteration doesn’t only apply to slugs and URLs in TYPO3. It also applies to file names, where UTF-8 encoding may still not be working.

  2. Paul Hansen commented <span data-processor="relativetime" data-value="2023-12-04T23:21:52+00:00" > 12/04/23, 23:21 </span>

    After some research, it looks like modern browsers will seamlessly handle percent-encoded URLs for more thorough l10n.

    Symfony supported this as of version 6.1 in 2022: https://symfony.com/blog/new-in-symfony-6-1-improved-routing-requirements-and-utf-8-parameters

    Just look around Wikipedia and you’ll see this in action. This doc provides some examples: https://developers.google.com/search/docs/crawling-indexing/url-structure

  3. Mathias Bolt Lesniak commented <span data-processor="relativetime" data-value="2023-12-05T02:17:40+00:00" > 12/05/23, 02:17 </span>

    Paul Hansen wrote in #note-29:
    > After some research, it looks like modern browsers will seamlessly handle percent-encoded URLs for more thorough l10n.

    Yes that is absolutely true. At the same time, it doesn’t remove the need for transliteration — especially where typing is involved. “That accented letter isn’t on my keyboard” or “how do I type 漢?”

    Mathias

  4. Garvin Hicking changed subject scandinavian letters are translittered wrong scandinavian letters are transliterated wrong Tags translit,transliteration,transliterate,slugs,utf8,latin <span data-processor="relativetime" data-value="2024-07-01T15:10:13+00:00" > 07/01/24, 15:10 </span>

  5. Gerrit Code Review pushed patch set 1 for main to change 85574 status Needs Feedback Under Review <span data-processor="relativetime" data-value="2024-08-09T14:34:28+00:00" > 08/09/24, 14:34 </span>

  6. Gerald Rintisch commented <span data-processor="relativetime" data-value="2025-04-14T03:58:41+00:00" > 04/14/25, 03:58 </span>

    Thanks for all the work!

    I would really love to see this feature in v14. I am building a website for Sweden and needed to rebase this patch.
    It works and is a must-have to use TYPO3 in a Scandinavian country.
    Too many companies are using WP or C#-based systems (← not really open-source). To compete this feature is absolutely needed :)

  7. Gerald Rintisch commented <span data-processor="relativetime" data-value="2026-01-05T07:58:40+00:00" > 01/05/26, 07:58 </span>

    Hello!
    Are there plans to make this working for TYPO3 v14?

  8. Benni Mack commented <span data-processor="relativetime" data-value="2026-10-23T08:04:56+00:00" > 10/23/26, 08:04 </span>

    Gerald Rintisch wrote in #note-49:
    > Hello!
    > Are there plans to make this working for TYPO3 v14?

    I think we need to define hwere we want it – slugs, file names etc?

  9. Mathias Bolt Lesniak commented <span data-processor="relativetime" data-value="2026-10-23T08:35:49+00:00" > 10/23/26, 08:35 </span>

    Benni Mack wrote in #note-51:
    > Gerald Rintisch wrote in #note-49:
    > > Hello!
    > > Are there plans to make this working for TYPO3 v14?
    >
    > I think we need to define hwere we want it – slugs, file names etc?

    This is basically about replacing the transliteration library, so anywhere transliteration happens. Ideally, it should be possible to use different implementations. A lot of progress on this has been made as a part of https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574 . It’s currently blocked by the way tests are set up and the DataHandler.

    I’m happy to jump back into this at any time.