GB 18030

From Vero - Wikipedia
Jump to navigation Jump to search

Template:Short description Template:More citations needed Template:Infobox character encoding

GB 18030 is a Chinese government standard, described as Information Technology — Chinese coded character set and defines the required language and character support necessary for software in China. GB18030 is the registered Internet name for the official character set of the People's Republic of China (PRC) superseding GB2312.<ref name=IANA>Template:Cite web</ref> As a Unicode Transformation FormatTemplate:Efn (i.e. an encoding of all Unicode code points), GB18030 supports both simplified and traditional Chinese characters. It is also compatible with legacy encodings including GB/T 2312, CP936,Template:Efn and GBK 1.0.

The Unicode Consortium has warned implementers that the latest version of this Chinese standard, GB 18030-2022, introduces what they describe as "disruptive changes" from the previous version GB 18030-2005 "involving 33 different characters and 55 code positions".<ref>Template:Cite web</ref> GB 18030-2022 was enforced from 1 August 2023.<ref>Template:Cite web</ref> It has been implemented in ICU 73.2; and in Java 21,<ref>Template:Cite web</ref> and backported to older Java 8, 11, 17 (LTS releases) and 20.0.2.<ref>Template:Cite web</ref>

In addition to the encoding method, this standard contains requirements about which additional scripts and languages should be represented, and to whom this standard is applicable.<ref name="CESI-FAQ">Template:Cite web Alt URL</ref> This standard however does not define the official character forms for the Chinese characters; this is standardised in List of Commonly Used Standard Chinese Characters.

History

Template:See also The GB18030 character set is formally called "Chinese National Standard GB 18030-2005: Information Technology—Chinese coded character set". GB abbreviates Template:Transliteration (Template:Lang), which means national standard in Chinese. The standard was published by the China Standard Press, Beijing, 8 November 2005. Only a portion of the standard is mandatory.<ref name="CESI-FAQ"/> Since 1 May 2006, support for the mandatory subset is officially required for all software products sold in the PRC.

Different Unicode mappings between GB 18030 versions
GB byte
sequence		 Unicode code point
GB 18030-2000 GB 18030-2005
A8 BC (ḿ) Template:Code Template:Unichar
81 35 F4 37 Template:Unichar Template:Code

An older version of the standard, known as "Chinese National Standard GB 18030-2000: Information Technology—Chinese ideograms coded character set for information interchange—Extension for the basic set", was published on March 17, 2000. The encoding scheme stays the same in the new version, and the only difference in GB-to-Unicode mapping is that GB 18030-2000 mapped the character Template:Code (ḿ) to a private use code point U+E7C7, and character Template:Code (without specifying any glyph) to U+1E3F (ḿ), whereas GB 18030-2005 swaps these two mapping assignments.<ref name="gb18030-2005">Template:Cite book</ref>Template:Rp More code points are now associated with characters due to update of Unicode, especially the appearance of CJK Unified Ideographs Extension B. Some characters used by ethnic minorities in China, such as Mongolian characters and Tibetan characters (GB 16959-1997 and GB/T 20542-2006), have been added as well, which accounts for the renaming of the standard.

Compared with its ancestors, GB 18030's mapping to Unicode has been modified for the 81 characters that were provisionally assigned a Unicode Private Use Area code point (U+E000–F8FF) in GBK 1.0 and that have later been encoded in Unicode.<ref>Template:Cite web</ref> This is specified in Appendix E of GB 18030.<ref name="gb18030-2005"/>Template:Rp<ref name="gb18030-2000">Template:Cite book</ref>Template:Rp There are 24 characters in GB 18030-2005 that are still mapped to Unicode PUA.<ref name=Lunde2006>Template:Cite web</ref>

In the GB 18030-2022 update, the requirements for characters to be mapped to PUA has been lifted completely and all characters should be mapped to their standard Unicode codepoints. Of these, 18 mappings were updated by position-swapping similar to what happened between GBK and GB 18030. The remaining six kept the two-byte PUA mappings, so that a change to the 4-byte sequence is needed to follow the non-PUA preference.<ref name="gb18030-2022-kenlunde">Template:Cite web</ref>

Template:AnchorPrivate use characters in GB-to-Unicode mappings
GB byte
sequence		 Unicode code point Template:Ref label
GBK 1.0<ref>Template:Cite web</ref><ref name="gb18030-2005"/>Template:Rp GB 18030-2005<ref name=Lunde2006/> Unicode 4.1 GB 18030-2022<ref name="gb18030-2022-kenlunde"/>
A6 D9<ref name="cjkv-info-proc">Template:Cite book</ref>Template:Rp U+E78D Template:Unichar
A6 DA U+E78E Template:Unichar
A6 DB U+E78F Template:Unichar
A6 DC U+E790 Template:Unichar
A6 DD U+E791 Template:Unichar
A6 DE U+E792 Template:Unichar
A6 DF U+E793 Template:Unichar
A6 EC U+E794 Template:Unichar
A6 ED U+E795 Template:Unichar
A6 F3 U+E796 Template:Unichar
A8 BC U+E7C7 Template:Unichar
A8 BF U+E7C8 Template:Unichar
A9 89 U+E7E7 Template:Unichar
A9 8A U+E7E8 Template:Unichar
A9 8B U+E7E9 Template:Unichar
A9 8C U+E7EA Template:Unichar
A9 8D U+E7EB Template:Unichar
A9 8E U+E7EC Template:Unichar
A9 8F U+E7ED Template:Unichar
A9 90 U+E7EE Template:Unichar
A9 91 U+E7EF Template:Unichar
A9 92 U+E7F0 Template:Unichar
A9 93 U+E7F1 Template:Unichar
A9 94<ref name="cjkv-info-proc"/>Template:Rp U+E7F2 Template:Unichar
A9 95 U+E7F3 Template:Unichar
FE 50 U+E815 Template:Unichar
FE 51 U+E816 Template:UnicharTemplate:Ref label U+E816
FE 52 U+E817 Template:UnicharTemplate:Ref label U+E817
FE 53 U+E818 Template:UnicharTemplate:Ref label U+E818
FE 54 U+E819 Template:Unichar
FE 55 U+E81A Template:Unichar
FE 56 U+E81B Template:Unichar
FE 57 U+E81C Template:Unichar
FE 58 U+E81D Template:Unichar
FE 59 U+E81E Template:Unichar
FE 5A U+E81F Template:Unichar
FE 5B U+E820 Template:Unichar
FE 5C U+E821 Template:Unichar
FE 5D U+E822 Template:Unichar
FE 5E U+E823 Template:Unichar
FE 5F U+E824 Template:Unichar
FE 60 U+E825 Template:Unichar
FE 61 U+E826 Template:Unichar
FE 62 U+E827 Template:Unichar
FE 63 U+E828 Template:Unichar
FE 64 U+E829 Template:Unichar
FE 65 U+E82A Template:Unichar
FE 66 U+E82B Template:Unichar
FE 67 U+E82C Template:Unichar
FE 68 U+E82D Template:Unichar
FE 69 U+E82E Template:Unichar
FE 6A U+E82F Template:Unichar
FE 6B U+E830 Template:Unichar
FE 6C U+E831 Template:UnicharTemplate:Ref label U+E831
FE 6D U+E832 Template:Unichar
FE 6E U+E833 Template:Unichar
FE 6F U+E834 Template:Unichar
FE 70 U+E835 Template:Unichar
FE 71 U+E836 Template:Unichar
FE 72 U+E837 Template:Unichar
FE 73 U+E838 Template:Unichar
FE 74 U+E839 Template:Unichar
FE 75 U+E83A Template:Unichar
FE 76 U+E83B Template:UnicharTemplate:Ref label U+E83B
FE 77 U+E83C Template:Unichar
FE 78 U+E83D Template:Unichar
FE 79 U+E83E Template:Unichar
FE 7A U+E83F Template:Unichar
FE 7B U+E840 Template:Unichar
FE 7C U+E841 Template:Unichar
FE 7D U+E842 Template:Unichar
FE 7E U+E843 Template:Unichar
FE 80 U+E844 Template:Unichar
FE 81 U+E845 Template:Unichar
FE 82 U+E846 Template:Unichar
FE 83 U+E847 Template:Unichar
FE 84 U+E848 Template:Unichar
FE 85 U+E849 Template:Unichar
FE 86 U+E84A Template:Unichar
FE 87 U+E84B Template:Unichar
FE 88 U+E84C Template:Unichar
FE 89 U+E84D Template:Unichar
FE 8A U+E84E Template:Unichar
FE 8B U+E84F Template:Unichar
FE 8C U+E850 Template:Unichar
FE 8D U+E851 Template:Unichar
FE 8E U+E852 Template:Unichar
FE 8F U+E853 Template:Unichar
FE 90 U+E854 Template:Unichar
FE 91 U+E855 Template:UnicharTemplate:Ref label U+E855
FE 92 U+E856 Template:Unichar
FE 93 U+E857 Template:Unichar
FE 94 U+E858 Template:Unichar
FE 95 U+E859 Template:Unichar
FE 96 U+E85A Template:Unichar
FE 97 U+E85B Template:Unichar
FE 98 U+E85C Template:Unichar
FE 99 U+E85D Template:Unichar
FE 9A U+E85E Template:Unichar
FE 9B U+E85F Template:Unichar
FE 9C U+E860 Template:Unichar
FE 9D U+E861 Template:Unichar
FE 9E U+E862 Template:Unichar
FE 9F U+E863 Template:Unichar
FE A0 U+E864 Template:Unichar
Notes
a.Template:NoteBlue indicates private use area
b.Template:NoteTemplate:Unichar mapped to 0x95329031 in GB 18030-2022
c.Template:NoteTemplate:Unichar mapped to 0x95329033 in GB 18030-2022
d.Template:NoteTemplate:Unichar mapped to 0x95329730 in GB 18030-2022
e.Template:NoteTemplate:Unichar mapped to 0x9536B937 in GB 18030-2022
f.Template:NoteTemplate:Unichar mapped to 0x9630BA35 in GB 18030-2022
g.Template:NoteTemplate:Unichar mapped to 0x9635B630 in GB 18030-2022

As a national standard

The first version of GB 18030, designated GB 18030-2000 Information Technology—Chinese coded character set for information interchange — Extension for the basic set, consists of 1-byte and 2-byte encodings, together with 4-byte encoding for CJK Unified Ideographs Extension A matching those in Unicode 3.0. The corresponding Unicode code points of this subset, including provisional private assignments, lie entirely in the BMP. These parts are fully mandatory in GB 18030-2000.<ref name="CESI-FAQ"/>Template:Rp Most major computer companies had already standardized on some version of Unicode as the primary format for use in their binary formats and OS calls. However, they mostly had only supported code points in the BMP originally defined in Unicode 1.0, which supported only 65,536 codepoints and was often encoded in 16 bits as UCS-2. This standard is basically an extension based on GBK with additional characters in CJK Unified Ideographs Extension A.

The second version designated GB 18030-2005 Information Technology—Chinese coded character set has the same mandatory subset as GB 18030-2000 of 1-, 2- and 4-byte encodings.<ref name="gb18030-2005" />Template:Rp This version also includes the full CJK Unified Ideographs Extension B in the 4-byte encoding section which is outside the BMP<ref name="Lunde2006" /> as a suggestion support requirement.<ref>Template:Cite web</ref> However, as the inclusion of CJK Unified Ideographs Extension B in a 4-byte region is required to be maintained during information processing, software can no longer get away with treating characters as 16-bit fixed width entities (UCS-2). Therefore, they must either process the data as a variable-width format (as with UTF-8 or UTF-16), which is the most common choice, or move to a larger fixed-width format (i.e. UTF-32). Microsoft made the change from UCS-2 to UTF-16 with Windows 2000. This version matches with Unicode 3.1, and also provided support for Hangul (Korean), Mongolian (including Manchu, Clear script, Sibe hergen, Galik), Tai Nuea, Tibetan, Uyghur/Kazakh/Kyrgyz and Yi.

The third and latest version, GB 18030-2022 Information Technology—Chinese coded character set, mandates the suggestion support part of CJK Unified Ideographs Extension B in GB 18030-2005, along with updates up to Unicode 11.0 including Kangxi Radicals and CJK Unified Ideographs URO, Extension C, D, E and F. Additional languages are also recognized by GB 18030-2022 such as part of Arabic, Tai Le, New Tai Lue, Tai Tham, Lisu, and Miao. GB 18030-2022 also introduces three implementation levels, with the requirement of "all products using this standard should implements Implementation Level 1" that includes 66 new BMP characters in the 4-byte encoding region that were added between Unicode 3.1 and Unicode 11.0. Implementation Level 2 requires the support of List of Commonly Used Standard Chinese Characters, and Implementation Level 3 requires all other specified regions in the standard.<ref name="gb18030-2022-kenlunde" />

Template:Main

From late 2022 to 2023, drafts of a further amendment are to be made to GB 18030-2022 available for public consultation. The current draft updates up to Unicode 15.1 on Ideographic Description Characters, CJK Unified Ideographs URO, Extension A, B, C, G, H and I.<ref>Template:Cite web</ref><ref>Template:Cite web</ref><ref name="irgn2623">Template:Cite web</ref> Originally, in late 2022, it would have placed 897 new sinographic characters in Plane 10 (hexadecimal: 0A), a yet-untitled astral Unicode plane, for citizen real-name certification in China, but eventually the repertoire (reduced to 622 characters after expert review) was fast-tracked into Unicode 15.1 in September 2023, as the CJK Unified Ideographs Extension I block.<ref name="wg2-n5222">Template:Cite web</ref> Following this, the amendment draft was modified to use the Extension I code points.<ref name="irgn2623" />

Mapping Template:Anchor

GB 18030 defines a one (ASCII), two (extended GBK), or four-byte (UTF) encoding. The two-byte codes are defined in a lookup table, while the four-byte codes are defined sequentially (hence algorithmically) to fill otherwise unencoded parts in UCS. GB 18030 inherits the bad aspects of GBK, most notably needing special code to safely find ASCII characters in a GB18030 sequence.

GB 18030 encoding<ref name="gb18030-2005"/>Template:Rp<ref name="gb18030-2000"/>Template:Rp<ref name=ICU>Authoritative mapping table between GB18030-2000 and Unicode. ICUTemplate:Snd International Components for Unicode. 2001-02-21. Accessed 2016-09-04.</ref>
GB 18030 code pointsTemplate:Efn Unicode
byte 1 (MSB) byte 2 byte 3 byte 4
Template:Code – Template:Code 128 Template:Code – Template:Code
Template:Code — invalidTemplate:Efn
Template:Code – Template:Code Template:Code – Template:Code except Template:CodeTemplate:Efn Template:Val Template:Code – Template:Code except Template:Code – Template:CodeTemplate:Efn
Template:Code – Template:Code Template:Code – Template:Code Template:Code – Template:Code Template:Code – Template:Code Template:Val
Template:Code — (Template:Val) reserved for future character extension
Template:Code – Template:Code — (Template:Val) reserved for future ideographic extension
unassigned — Template:Code – Template:CodeTemplate:Efn
Template:Code – Template:Code Template:Code – Template:Code Template:Code – Template:Code Template:Code – Template:Code Template:Val Template:Digit groups – Template:Digit groups
Template:Code – Template:Code — (Template:Val) reserved for future standard extension
Template:Code – Template:Code — (Template:Val) user-defined
Template:Code — invalid
Total Template:Val

The one- and two-byte code points are essentially GBK with the euro sign, PUA mappings for unassigned/user-defined points, and vertical punctuations. The four byte scheme can be thought of as consisting of two units, each of two bytes. Each unit has a similar format to a GBK two byte character but with a range of values for the second byte of 0x30–0x39 (the ASCII codes for decimal digits). The first byte has the range 0x81 to 0xFE, as before. This means that a string-search routine that is safe for GBK should also be reasonablyTemplate:Clarify safe for GB18030 (in much the same way that a basic byte-oriented search routine is reasonably safe for EUC).

This gives a total of 1,587,600 (126×10×126×10) possible 4-byte sequences, which is easily sufficient to cover Unicode's 1,112,064 (17×65536 − 2048 surrogates) assigned, reserved, and noncharacter code points.

Unfortunately, to further complicate matters there are no simple rules to translate between a 4-byte sequence and its corresponding code point. Instead, codes are allocated sequentially (with the first byte containing the most significant part and the last the least significant part) only to Unicode code points that are not mapped in any other manner.Template:Efn For example: 

U+00DE (Þ) → 81 30 89 37
U+00DF (ß) → 81 30 89 38
U+00E0 (à) → A8 A4
U+00E1 (á) → A8 A2
U+00E2 (â) → 81 30 89 39
U+00E3 (ã) → 81 30 8A 30

An offset table is used in the WHATWG and W3C version of GB 18030 to efficiently translate code points.<ref name=whatwg>Template:Cite web</ref> ICU<ref name=ICU/> and glibc use similar range definitions to avoid wasting space on large sequential blocks.

Support

Template:Anchor Template:Expand section

Encoding

GBTemplate:Nbsp18030 has been supported on Windows since the release of Windows 95, as code page 54936.<ref>Template:Cite web</ref> Windows 2000 and XP offer a GB18030 Support Package.<ref>Template:Cite web</ref> The open source PostgreSQL database supports GB18030 through its full support for UTF-8, i.e. by converting it to and from UTF-8. Similarly Microsoft SQL Server supports GB18030 by conversion to and from UTF-16.

More specifically, supporting the GB18030 encoding on Windows means that Code Page 54936 is supported by MultiByteToWideChar and WideCharToMultiByte. Due to the backward compatibility of the mapping, many files in GB18030 can be actually opened successfully as the legacy Code Page 936, that is GBK, even if the Code Page 54936 is not supported. However, that is only true if the file in question contains only GBK characters. Loading will fail or cause corrupted result if the file contains characters that do not exist in GBK (see Template:Slink for examples).

GNU glibc's gconv, the character codec library used on most Linux distributions, supports GB 18030-2000 since 2.2,<ref>Template:Cite web</ref> and GB 18030-2005 since 2.14;<ref>Template:Cite web</ref> glibc notably includes non-PUA mappings for GB 18030-2005 in order to achieve round-trip conversion.<ref>Template:Cite web</ref> GNU libiconv, an alternative iconv implementation frequently used on non-glibc UNIX-like environments like Cygwin, supports GB 18030 since version 1.4.<ref>Template:Cite web</ref>

As of 2022, "supporting non-Chinese scripts continues to be optional"<ref name="non-Chinese">Template:Cite web</ref> (presumably for display/font support only; and in China, since the encoding is a full UTF). The standard is known to support English/ASCII and the "following non-Chinese scripts are recognized by GB 18030-2022: Arabic, Tibetan, Mongolian, Tai Le, New Tai Lue, Tai Tham, Yi, Lisu, Hangul (Korean), and Miao."<ref name="non-Chinese" />

Fonts

The GB18030 Support Package for Windows XP and Windows 2000 contained SimSun18030.ttc, a TrueType font collection file which combines two Chinese fonts, SimSun-18030 and NSimSun-18030. The SimSun-18030 font included all the mandatory CJK characters required in GB18030-2000, including characters in the CJK Unified Ideographs and CJK Unified Ideographs Extension A blocks of Unicode 3.0. Since Windows Vista, Windows has included the Simsun and Simsun-ExtB fonts. (Simsun supports CJK characters in the Unicode Basic Multilingual Plane, while Simsun-ExtB supports most CJK characters in the Unicode Supplementary Ideographic Plane). These fonts have been updated in subsequent Windows releases to support additional CJK characters. As of 2022, the Simsun-ExtB font supported CJK Unified Ideograph Extension B, Extension C and Extension D. For GB 18030-2022, the Simsun-ExtB font was updated to add support for CJK Unified Ideograph Extension E and Extension F.<ref name="KB5028171"/> Also, the Simsun-ExtG font was added to Windows 10 and Windows 11 to support CJK characters in CJK Unified Ideographs Extension G, Extension H and Extension I.<ref>Template:Cite web</ref><ref>Template:Cite web</ref> The Simsun fonts in Windows 10 and Windows 11 support all CJK characters required for GB18030-2022 plus Amendment 1 level 3 conformance.

Microsoft YaHei and DengXian provided by Microsoft were updated in 2023 to match GB 18030-2022 implementation level 2.<ref name="KB5028171">Template:Cite web</ref>

Source Han Sans (and its counterpart Noto Sans CJK) are already compliant with GB 18030-2022 implementation level 2 when the standard update for GB 18030 is announced Template:As of. Source Han Serif (and its counterpart Noto Serif CJK) however is not compliant at the time, and an update is provided to ensure the font is compliant to implementation level 2. Similarly Microsoft YaHei and PingFang (Apple) require a small number of URO additions that are associated with implementation level 1 in order to become compliant with GB 18030-2022 implementation level 2.<ref name="non-Chinese" />

Other CJK font families like HAN NOM<ref>Template:Cite web</ref> and Hanazono Mincho<ref>Template:Cite web</ref> provide wider coverage for Unicode CJK Extension blocks than SimSun-18030 or even SimSun (Founder Extended), but they don't support all code points defined in GB 18030.

See also

Notes

Template:Notelist

References

Template:Reflist

Template:CJK computing Template:Character encoding Template:Unicode navigation

Retrieved from "https://wiki.sarg.dev/index.php?title=GB_18030&oldid=302938"