"A string is an immutable ordered sequence of 16-bit values, each of which typically represents a Unicode character."

The code points in each plane have the hexadecimal values xx0000 to xxFFFF, where xx is a hex value from 00 to 10, signifying which plane the values belong to.

The first plane (code points U+0000 to U+FFFF) contains the most frequently used characters and is called the Basic Multilingual Plane or BMP. UTF-16 encodes valid code points in this range as single 16-bit code units that are numerically equal to the corresponding code points.

e is 1 character with 17-bit codepoint 0x1d452

Character encodings are one of the many things I've studied a lot over the years, because writing code used in asian countries is something that understanding of has been of real practical value.

However, there's an awful lot of computing history tied up in and expressed in character encodings; there are a lot of nuances to character encodings which come from accidents of history, and so to really understand how things are now you have to walk quite a long road from computing's early days.

In particular, there's an oft-assumed correspondence between the notion of an 8-bit "byte" as the unit of character encoding, and as the fundamental addressable unit in computer memory. However, these aren't quite the same thing at all. IBM mainframes just as the IBM 360 happened to be 32-bit machines, and IBM developed a character encoding to fit within the 32-bit machine word by packing 4 8-bit characters together, leading to the EBCDIC encoding (which had things like upper and lower case).


Later on, DEC had machines which happened to have a 36-bit machine word, and used a character encoding of just six bits per character to be able to pack six characters per machine word and CPU register, and later on again the DEC PDP-11 appeared - which was a 16-bit machine, but one in which the memory was able to be addressed in a "half" unit of 8 bits, and for which a programming language - C - has developed that mapped the notion of character to that. For these machines, the SIXBIT encoding of the earlier machine words was expanded to a 7-bit encoding (with one bit left spare) that formed the basis of first the US ASCII standard and then a host of global standards (under ISO 646) derived from it.

Almost all the early Internet protocols and standards were written with an eye to the difference between characters and bytes because almost all machines at the time had different ideas about these things; so, textual data would be sent between machines that used radically different encodings an numbers of bits per character. It so happened that the microprocessor era - arriving after the PDP-11 - fixed on 8 bits as the fundamental unit of everything and so over the last 30 years that has become universal, but it wasn't always so.

So, some background. Throughout the 1980's, especially as UNIX and C on microprocessors spread into Asian countries and most notably Japan, the problem of dealing with very large character sets became of practical importance. However, there were 3 fronts to this:

a) The problem of data processing within systems; the languages and tools of the microprocessor era, especially C, did not make any distinction between a character and a byte. The ISO 646 national standards proliferated by mostly switching between incompatible single-byte encodings, but for Asian languages there was no choice but to encode characters as variable-length sequences of bytes; this broke the correspondence between byte = character fundamentally and meant that many programs did not work on non-US characters at all.

b) The problem of data interchange between systems; to even move data in single-byte encodings from machine to machine, you needed to know which of the national variant formats it was encoded in so it could be successfully interpreted.

c) The problem of meaningfully displaying text; bitmapped displays were on the rise, so that more and more different display forms of characters were becoming possible, but many asian and arabic languages there is not a direct correspondence between how things display (their visual form) and the actual language form or encoding, not just in terms of differences in reading direction (some languages are right-to-left or bottom-to-top) but in other ways too - in written Thai, vowels in the language are represented as tone marks above and below consonants, while Arabic uses a relatively small alphabet but is rendered using an incredible variety of ligatured forms so that character display is context-sensitive.

So, in the 80's several things happened; a) and b) in combination lead to pursuing a single 16-bit "universal" character encoding which could represent basically every active natural language which had been through the ISO 646 process, so that the interchange and processing problem could be solved by transcoding from separate national encodings (some of which required multi-byte sequences, as e.g. Japanese) through the wider 16-bit "universal" encoding. An industry consortium, called the Unicode Foundation, was formed to create the encoding.

At the same time, the ISO bodies wanted to standardize computer languages, and so a huge part of the ISO effort for C which culminated in the 1989 standard was to add machinery to the language to require support for multi-byte character encodings so that programs could be reliably written for the existing ISO 646 encodings, and also to add support for things like the 16-bit "Universal" encoding of Unicode, although they hedged their bets about what the exact universal "wide" encoding would be.

As for c), one of the things that Unicode tried to deal with and force people to face was the difference between the notion of an encoded character, which was important to standardize for data interchange, and that is was different to the notion of what a character looked like (which we call a glyph). Programmers of all stripes tended to assume because western languages had a one-to-one correspondence between these things that they were the same concept, whereas they really are different things in other languages. So, Unicode carefully specified the encodings, but made a big deal of the fact that glyphs were a different thing altogether and that people should stop assuming they were.