UTF-8 Databases
UTF-8 Databases
Docs
PHPFusion 8 Documentation
IntroductionFeatures ListRequirementsInstallationUpgradingContent AdminUser AdminSystem AdminSettings Admin Contributors
Falk added UTF-8 Databases on 08/11/2014
Falk added UTF-8 Databases on
Falk edited UTF-8 Databases on 26/10/2020
UTF-8 Databases
PHPFusion are using utf8mb4_unicode_ci for it´s Database encodings as default from PHPFusion 9.0.
Accuracy of sorting
utf8mb4_unicode_ci is based on the Unicode standard for sorting, and sorts accurately in a very wide range of languages.
utf8_general_ci comes close to correct Unicode sorting in many common languages, but has a number of limitations: in some languages, it won't sort correctly at all.
In others, it will merely have some quirks.
Performance
utf8_general_ci is faster at comparisons and sorting, because it takes a bunch of performance-related shortcuts.
utf8mb4_unicode_ci uses a much more complex comparison algorithm which aims for correct sorting according in a very wide range of languages.
This makes it slower to sort and compare large numbers of fields.
Unicode defines complex sets of rules for how characters should be sorted. These rules need to take into account language-specific conventions; not everybody sorts their characters in what we would call 'alphabetical order'.
As far as Latin (ie "European") languages go, there is not much difference between the Unicode sorting and the simplified utf8_general_ci sorting in MySQL, but there are still a few differences:
For examples, the Unicode collation sorts "ß" like "ss", and "Œ" like "OE" as people using those characters would normally want, whereas utf8_general_ci sorts them as single characters (presumably like "s" and "e" respectively).
In non-latin languages, such as Asian languages or languages with different alphabets, there may be a lot more differences between Unicode sorting and the simplified utf8_general_ci sorting.
The suitability of utf8_general_ci will depend heavily on the language used. For some languages, it'll be quite inadequate.
Some Unicode characters are defined as ignorable, which means they shouldn't count toward the sort order and the comparison should move on to the next character instead. utf8mb4_unicode_ci handles these properly.
What should you use?
There is almost never any reason to use utf8mb4_unicode_ci anymore, as we have left behind the point where CPU speed is low enough that the performance difference would be important. Your database will almost certainly be limited by quite other bottlenecks than this nowadays. The difference in performance is only going to be measurable in extremely specialized situations, and if that's you, you'd already know about it. If you're experiencing slow sorting, in almost all cases it'll be an issue with your indexes/query plan. Changing your collation function should not be high on the list of things to troubleshoot.
It's more important to sort properly in whichever language your users are using.
Even if you know your application only supports the English language, it may still need to deal with people's names, which can often contain characters used in other languages in which it is just as important to sort correctly.
Using the Unicode rules for everything helps add peace of mind that the very smart Unicode people have worked very hard to make sorting work properly.
Accuracy of sorting
utf8mb4_unicode_ci is based on the Unicode standard for sorting, and sorts accurately in a very wide range of languages.
utf8_general_ci comes close to correct Unicode sorting in many common languages, but has a number of limitations: in some languages, it won't sort correctly at all.
In others, it will merely have some quirks.
Performance
utf8_general_ci is faster at comparisons and sorting, because it takes a bunch of performance-related shortcuts.
utf8mb4_unicode_ci uses a much more complex comparison algorithm which aims for correct sorting according in a very wide range of languages.
This makes it slower to sort and compare large numbers of fields.
Unicode defines complex sets of rules for how characters should be sorted. These rules need to take into account language-specific conventions; not everybody sorts their characters in what we would call 'alphabetical order'.
As far as Latin (ie "European") languages go, there is not much difference between the Unicode sorting and the simplified utf8_general_ci sorting in MySQL, but there are still a few differences:
For examples, the Unicode collation sorts "ß" like "ss", and "Œ" like "OE" as people using those characters would normally want, whereas utf8_general_ci sorts them as single characters (presumably like "s" and "e" respectively).
In non-latin languages, such as Asian languages or languages with different alphabets, there may be a lot more differences between Unicode sorting and the simplified utf8_general_ci sorting.
The suitability of utf8_general_ci will depend heavily on the language used. For some languages, it'll be quite inadequate.
Some Unicode characters are defined as ignorable, which means they shouldn't count toward the sort order and the comparison should move on to the next character instead. utf8mb4_unicode_ci handles these properly.
What should you use?
There is almost never any reason to use utf8mb4_unicode_ci anymore, as we have left behind the point where CPU speed is low enough that the performance difference would be important. Your database will almost certainly be limited by quite other bottlenecks than this nowadays. The difference in performance is only going to be measurable in extremely specialized situations, and if that's you, you'd already know about it. If you're experiencing slow sorting, in almost all cases it'll be an issue with your indexes/query plan. Changing your collation function should not be high on the list of things to troubleshoot.
It's more important to sort properly in whichever language your users are using.
Even if you know your application only supports the English language, it may still need to deal with people's names, which can often contain characters used in other languages in which it is just as important to sort correctly.
Using the Unicode rules for everything helps add peace of mind that the very smart Unicode people have worked very hard to make sorting work properly.