荔园在线
荔园之美，在春之萌芽，在夏之绽放，在秋之收获，在冬之沉淀
[回到开始]
[上一篇][下一篇]
发信人: tang (独孤九剑〖玄铁重剑〗), 信区: Program
标 题: Draft Unicode FAQ
发信站: BBS 荔园晨风站 (Tue Feb 15 15:06:30 2000), 站内信件
Draft Unicode FAQ
Revision 2
Authors Mark Davis (mark@unicode.org)
Date 1999-05-17
This Version http://www.unicode.org/unicode/faq/index-2.html
Previous Version n/a
Latest Version http://www.unicode.org/unicode/faq/
Summary
This document provides answers to a number of commonly asked questions that
software developers have about Unicode. It is only a supplement to the Unicode
Standard--for complete information you should always consult the standard
itself, including updates on the web. The topics are divided up into the
following sections:
Basic Questions
Characters
Directionality
Languages and CJK
Case Mappings
Encoding Forms
UTF-16 and UCS-4
BOM
Unicode and ISO 10646
Status
While this document needs further editing and revision, the Unicode editorial
committee decided it would be better to make it available even in a preliminary
form, so that public feedback can be used in developing the further versions.
Acknowledgements
Many of the Q&A's were contributed by different volunteers. The source is
acknowledged where possible, by inclusion in square brackets at the end of the
answer. However, the original contribution may have been edited during
incorporation into this document. If you have a Q&A that you would like to see
added, please send it to the author (note that not all such submissions will be
included). Make sure that the subject header starts with either "FAQ" or "RE:
FAQ".
Ken Whistler, Joe Becker, Lloyd Honomichl, Paul Hoffman, Jim Agenbroad, and
Markus Scherer also contributed to the review of this material.
--------------------------------------------------------------------------------
Basic Questions
Q: In the past, we have just handed off our code to a translation agency. What'
s wrong with that?
Often, companies develop a first version of a program or system to just deal
with English. When it comes time to produce a first international version, a
common tactic is to just go through all the lines of code, and translate the
literal strings.
While this may work once, it is not a pattern that you want to follow. Not all
literal strings get translated, so this process requires human judgment, and is
time-consuming. Each new version is expensive, since people have to go through
the same process of identifying the strings that need to be changed. In
addition, since there are multiple versions of the source code, maintenance and
support becomes expensive. Moreover, there a high risk that a translator may
introduce bugs by mistakenly modifying code.
Q: What is the modern approach?
The general technique used now is to internationalize the programs. This means
to prepare them so that the code never needs modification--separate files
contain the translatable information. This involves a number of modifications
to the code:
move all translatable strings into separate files called resource files, and
make the code access those strings when needed. These resource files can be
flat text files, databases, or even code resources, but they are completely
separate from the main code, and contain nothing but the translatable data.
change variable formatting to be language-independent. This means that dates,
times, numbers, currencies, and messages all call functions to format according
to local language and country requirements.
change sorting, searching and other types of processing to be
language-independent.
Once this process is concluded, you have an internationalized program. To
localize that program then involves no changes to the source code. Instead,
just the translatable files are typically handed off to contractors or
translation agencies to modify. The initial cost of producing internationalized
code is somewhat higher than localizing to a single market, but you only pay
that once. The costs of simply doing a localization, once your code is
internationalized, is a fraction of the previous cost--and avoids the
considerable cost of maintenance and source code control for multiple code
versions.
Q: How does Unicode play in internationalization?
Unicode is the new foundation for this process of internationalization. Older
codepages were difficult to use, and have inconsistent definitions for
characters. Internationalizing your code while using the same code base is
complex, since you would have to support different character sets--with
different architectures--for different markets.
But modern business requirements are even stronger; programs have to handle
characters from a wide variety of languages at the same time: the EU alone
requires several different older character sets to cover all its languages.
Mixing older character sets together is a nightmare, since all data has to be
tagged, and mixing data from different sources is nearly impossible to do
reliably.
With Unicode, a single internationalization process can produce code that
handles the requirements of all the world markets at the same time. Since
Unicode has a single definition for each character, you don't get data
corruption problems that plague mixed codeset programs. Since it handles the
characters for all the world markets in a uniform way, it avoids the
complexities of different character code architectures. All of the modern
operating systems, from PCs to mainframes, support Unicode now or are actively
developing support for it. The same is true of databases, as well.
Q: What was wrong with using classical character sets for application programs?
Different character sets have very different architectures. In many, even
simply detecting which bytes form a character is a complex,
contextually-dependent process. That means either having multiple versions of
the program code for different markets, or making the program code is much,
much more complicated. Both of these choices involve development, testing,
maintenance, and support problems. These make the non-US versions of programs
more expensive, and delay their introduction, causing significant loss of
revenue.
Q: What was wrong with using classical character sets for databases?
Classical character sets only handle a few languages at a time. Mixing
languages was very difficult or impossible. In today's markets, mixing data
from many sources all around the world, that strategy for products fails badly.
The code for a simple letter like "A" will vary wildly between different sets,
making searching, sorting, and other operations very difficult. There is also
the problem of tagging every piece of textual data with a character set, and
corruption problems when mixing text from different character sets.
Q: What is different about Unicode?
Unicode provides a unique encoding for every character. Once your data is in
Unicode, it can be all handled the same way--sorted, searched, and manipulated
without fear of data corruption.
Q: You talk about Unicode being the right technical approach. But is it being
accepted by the market?
The industry is converging on Unicode for all internationalization. For example,
Microsoft NT is built on a base of Unicode; AIX, Sun, HP/UX all offer Unicode
support. All the new web standards; HTML, XML, etc. are supporting or requiring
Unicode. The latest versions of Netscape Navigator and Internet Explorer both
support Unicode. Sybase, Oracle, DB2 all offer or are developing Unicode
support.
Most significant application programs with international versions either
support Unicode or are moving towards it. For example, Microsoft's products are
rapidly being adapted to use Unicode: most of Office 98 is Unicode-capable.
This is a good illustration--Microsoft first started by merging their East
Asian (Chinese, Japanese, and Korean) plus their US version into a single
program using Unicode. They will be merging in their Middle East and South
Asian support in the future.
Q: What about the Far East support?
Unicode incorporates the characters of all the major government standards for
ideographic characters from Japan, Korea, China, and Taiwan, and more. The
Unicode Standard, Version 2.1 has more than 21,000 ideographic characters. The
Unicode Consortium actively works with the IRG committee of ISO SC2/WG2 to
define additional sets of ideographic characters for inclusion: The Unicode
Standard 3.0 has an additional 6,000 ideographic characters.
Q: So all I need is Unicode, right?
Unicode is not a magic wand; it is a standard for the storage and interchange
of textual data. Somewhere there has to be code that recognizes and provides
for the conventions of different languages and countries. These conventions can
be quite complex, and require considerable expertise to develop code for and to
produce the data formats. Changing conditions and new markets also require
considerable maintenance and development. Usually this support is provided in
the operating system, or with a set of code libraries.
Q: Unicode has all sorts of features: combining marks, bidirectionality,input
methods, surrogates, Hangul syllables, etc. Isn't a big burden to support?
Unicode by itself is not complicated to implement--it all depends on which
languages you want to support. The character repertoire you need fundamentally
determines the features you need to have for compliance. If you just want to
support Western Europe, you don't need to have much implementation beyond what
you have in ASCII.
Which further characters you need to support is really dependent on the
languages you want, and what system requirements you have (servers, for example,
may not need input or display). For example, if you need East Asian languages
(in input), you have to have input methods. If you support Arabic or Hebrew
characters (in display), then you need the bidirectional algorithm. For normal
applications, of course, much of this will be handled by the operating system
for you.
Q: What level of support should I look for?
Unicode support really divides up into two categories: server-side support and
client-side support. The requirements for Unicode support in these two
categories can be summarized as follows (although you may only need a subset of
these features for your projects):
Full server-side Unicode support consists of:
Storage and manipulation of Unicode strings.
Conversion facilities to a full complement of other charsets (8859-x, JIS,
EBCDIC, etc.)
A full range of formatting/parsing functionality for numbers, currencies,
date/time and messages for all locales you need.
Message cataloging (resources) for accessing translated text.
Unicode-conformant collation, normalization, and text boundary (grapheme, word,
line-break) algorithms.
Multiple locales/resources available simultaneously in the same application or
thread.
Charset-independent locales (all Unicode characters usable in any locale).
Full client-side support consists all the same features as server-side, plus
GUI support:
Displaying, printing and editing Unicode text.
This requires BIDI display if Arabic and Hebrew characters are supported.
This requires character shaping if scripts such as Arabic and Indic are
supported.
Inputting text (e.g. with Japanese input methods)
Full incorporation of these facilities into the windowing system and the
desktop interface.
Q: Are there on-line resources to help me with Unicode?
Yes. Here is a short table to help you find some of them.
Question Reference
What is in each particular version of Unicode?
What is the latest version of Unicode?
Versions of the Unicode Standard:
http://www.unicode.org/unicode/standard/versions/
How do I handle CRLF in Unicode?
What should regular expressions do with Unicode?
Can I transmit Unicode text on EBCDIC systems?
How should a word-processor break lines in Unicode text?
Are there ways to normalize Unicode text?
For the Far East, how do I decide which characters should use wide glyphs and
which ones narrow?
How should I sort Unicode text?
Is there an update to the BIDI algorithm?
How can I compress Unicode text?
Unicode Technical Reports:
http://www.unicode.org/unicode/reports/
I want to get online data for implementing Unicode. Where can I find data for:
Character properties?
Upper/lower/titlecasing?
Decompositions?
Normalization?
Conversion to other character encodings?
Code for Kanji code conversion with compressed tables?
Online Data:
http://www.unicode.org/unicode/onlinedat/
Are there conferences where we can find out more about Unicode?
Unicode Conferences:
http://www.unicode.org/unicode/conf.html
I am interested in joining the consortium. Where can I find out more?
Membership Information:
http://www.unicode.org/unicode/consortium/
Q: What does Unicode conformance require?
Chapter 3 discusses this in detail. Here's a very informal version:
1) Unicode code units are 16 bits long; deal with it.
2) Byte order is only an issue in files.
3) If you don't have a clue, assume big-endian.
4) Loose surrogates don't mean jack.
5) Neither do U+FFFE and U+FFFF.
6) Leave the unassigned codepoints alone.
7) It's OK to be ignorant about a character, but not plain wrong.
8) Subsets are strictly up to you.
9) Canonical equivalence matters.
10) Don't garble what you don't understand.
[John Cowan]
--------------------------------------------------------------------------------
Characters
Q: Does "text element" mean the same as "combining character sequence"?
No, this is a common misperception. A text element just means any sequence of
characters that are treated as a unit by some process (see page 2-2). A
combining character sequence is a base character followed by any number of
combining characters. It is one type of a text element, but words and sentences
are also examples of text elements.
Q: So is a combining character sequence the same as a "character"?
That depends. For a programmer, a Unicode code value represents a single
character (for exceptions, see below). For an end user, it may not. The better
word for what end-users think of as characters is grapheme (as defined on page
G-5): a minimally distinctive unit of writing in the context of a particular
writing system.
For example, ? (A + COMBINING RING or A-RING) is a grapheme in the Danish
writing system, while KA + VIRAMA + TA + VOWEL SIGN U is one in the Devanagari
writing system. Graphemes are not necessarily combining character sequences,
and combining character sequences are not necessarily graphemes. Moreover,
there are a number of other cases where a user would not count "characters" the
same way as a programmer would: where there are invisible characters such as
the RLM used in BIDI, compatibility composites such as "Dz", "ij", or Roman
numerals, and so on.
Q: I would think that the following characters (and probably a few others)
would have compatibility decompositions. Why don't they?
2044 (FRACTION SLASH) --> 002F (SOLIDUS)
2010 (HYPHEN) --> 002D (HYPHEN-MINUS)
2013 (EN DASH) --> 002D (HYPHEN-MINUS)
2014 (EM DASH) --> 002D 002D (HYPHEN-MINUS, HYPHEN-MINUS)
These are called "confusables": characters that look similar, but do have
distinct behavior and generally distinct appearance (length or angle). Consult
the Unicode Standard for descriptions of the differences between these
characters.
Compatibility characters are really just particular presentation forms of
another character (or sequence of characters), encoded to ensure round-trip
conversion to legacy encodings.
Q: Do all the compatibility ideographs have equivalents?
No, the ideographs FA0E, FA0F, FA11, FA13, FA14, FA1F, FA21, FA23, FA24, FA27,
FA28, and FA29 have no canonical equivalents. These 12 characters are not
duplicates and should be treated as a small extension of the set of unified
ideographs. In fact, they are derived from industry standards, but are not
duplicates of anything. They didn't make it into the main Unihan block because
they aren't in any preexisting national standard.
[John Cowan]
Q: Do all the Unicode character set mappings cover control codes?
No, the control code mappings are often omitted from the tables on the Unicode
site. For the ASCII family of character sets, these are usually one-to-one
mappings from the Unicode set based on taking the lower 8 bits of the Unicode
character. However, they may differ significantly for other sets, such as
EBCDIC.
The correct Unicode mappings for the special graphic characters (01-1F, 7F) of
CP437 and other DOS-type code pages are available at
ftp://unicode.org/pub/MAPPINGS/VENDORS/MISC/IBMGRAPH.TXT, not exactly an
obvious location. The rest of the DOS code pages are at
ftp://unicode.org/pub/MAPPINGS/VENDORS/MICSFT/PC.
[John Cowan]
Q: Is the POSIX ctype.h model sufficient for Unicode?
POSIX "ctype.h" knows but two cases, whereas Unicode knows three. In POSIX,
only European Arabic digits can pass "isdigit", whereas Unicode has many sets
of digits, all putatively equal. In POSIX "ctype.h", that which is "alnum" but
not "alpha" must be a "digit", but Unicode is aware that not all numbers are
digits, nor are all letters alphabetic. Unicode groks spacing and non-spacing
marks, but POSIX comprehends them not.
[John Cowan]
--------------------------------------------------------------------------------
Directionality
Q: What is meant by the "directionality" of a script?
Individual writing systems make different default assumptions about how
characters are arranged into lines and lines are arranged on a page or screen.
Such assumptions are referred to as a script's directionality. For example, in
the Latin script, characters run horizontally from left to right to form lines,
and lines run from top to bottom.
Semitic scripts arrange characters right-to-left into lines, although digits
run the other way, making the scripts inherently bidirectional. Ordering
characters into lines can be even more complex when left-to-right and
right-to-left scripts are used together. Because bidirectional scripts can have
opposite directions on the same line, and because the direction of punctuation
characters is determined by their surroundings, resolving the actual direction
of a specific part of the line depends on context analysis. The Unicode
Standard defines an implicit algorithm to determine the layout of a line, and
also provides overrides to handle situations that are ambiguous; see Section
3.11, Bidirectional behavior, for more information.
[John Jenkins]
Q: Are any scripts written vertically?
East Asian scripts are frequently written in vertical lines which run from
top-to-bottom and are arrange columns either from left-to-right (Mongolian) or
right-to-left (other scripts). Most characters use the same shape and
orientation when displayed horizontally or vertically, but many punctuation
characters will change their shape when displayed vertically.
Letters and words from other scripts are generally rotated through ninety
degree angles so that they, too, will read from top to bottom. That is, letters
from left-to-right scripts will be rotated clockwise and letters from
right-to-left scripts counterclockwise, both through ninety degree angles.
Unlike the bidirectional case, the choice of vertical layout is usually treated
as a formatting style; therefore, the Unicode Standard does not define default
rendering behavior for vertical text nor provide directionality controls
designed to override such behavior.
[John Jenkins]
Q: Are there any other script directions?
Other script directionalities are possible and are found in actual writing
systems, mainly in historical ones. For example, some ancient Numidian texts
are written bottom-to-top, and Egyptian hieroglyphics can be written with
arbitrary directions for individual lines.
One prominent example is boustrophedon (literally, "ox-turning"), which is
often found in ancient European writing systems such as early Greek. In
boustrophedon writing, characters are arranged into horizontal lines, but the
individual lines alternate between running right to left and running left to
right, the way an ox goes back and forth when plowing a field. The letters
themselves use mirrored images in accordance with each individual line's
direction.
[John Jenkins]
Q: So do developers need to worry about these historical directions?
Not really. Boustrophedon writing is of interest almost exclusively to scholars
intent on reproducing the exact visual content of ancient texts. The Unicode
Standard does not provide formatting codes to signal boustrophedon text.
Specialized word processors for ancient scripts might offer support for this.
In the absence of that, fixed texts can be written in boustrophedon by using
hard line breaks and directionality overrides
[John Jenkins]
--------------------------------------------------------------------------------
Languages and CJK
Q: Won't users be confused by CJK characters being presented in different font
styles for different countries?
It is true that some Unihan characters are typically written differently within
the Japanese, Taiwanese, Korean, and Mainland Chinese typographic traditions.
These differences of writing style are within the general range of allowable
differences within each typographic tradition.
E.g., the official "Taiwanese" glyph for U+8349 ("grass") per ISO/IEC 10646
uses four strokes for the "grass" radical, whereas the PRC, Japanese, and
Korean glyphs use three. As it happens, Apple's LiSung Light font for Big Five
(which follows the "Taiwanese" typographic tradition) uses three strokes. (This
is easily confirmed by accessing
http://charts.unicode.org/unihan/unihan.acgi$8349.)
Japanese users prefer to see Japanese text written with "Japanese" glyphs.
It is also acceptable to Japanese users to see Chinese text written with
"Japanese" glyphs. For example:
A standard Japanese dictionary which quotes Chinese authors (e.g., Mencius)
uses "Japanese" glyphs, not Chinese ones.
In particular, it is perfectly acceptable within Japanese typography for
stretches of Chinese quoted in a predominantly Japanese text to be written with
"Japanese" glyphs.
Han unification allows for the possibility that a Japanese user might be
required to use a Chinese font to display some Japanese text (e.g., if it uses
a rare kanji).
Ditto for JIS or an ISO 2022-based solution.
Unicode doesn't include all the characters in actual use in Japan today,
particularly for personal names.
Neither does JIS or an ISO 2022-based solution. There are vendor sets which
include many of these characters, and Unicode is working with the IRG and East
Asian national bodies to add them.
[John Jenkins]
--------------------------------------------------------------------------------
Case Mappings
Q: Do all scripts have upper and lower case?
No, as a matter of fact, most scripts do not have cases.
[Jonathan Rosenne]
Q: Do the case mappings in Unicode allow a round-trip?
No, there are instances where two characters map to the same result. For
example, both a sigma and a final sigma both uppercase to a capital sigma.
There are other cases where the uppercase of a character requires
decomposition. In some cases, the correct mapping also depends on the locale.
For example, in Turkish, an i maps to an uppercase dotted I.
Q: Doesn't this cause a problem?
Remember that in general, case mappings of strings lose information and thus do
not allow round tripping. Take the word "anglo-American" or the Italian word
"vederLa". Once you uppercase, lowercase or titlecase these strings, you can't
recover the original just by performing the reverse operation.
Q: Why aren't there extra characters to support locale-independent casing for
Turkish?
The fact is that there is too much data coded in 8859-9 (with 0xDD = LATIN
CAPITAL LETTER I WITH DOT and 0xFD = LATIN SMALL LETTER DOTLESS I) which
contains both Turkish and non-Turkish text. Transcoding this data to Unicode
would be intolerably difficult if it all had to be tagged first to sort out
which 0x49 characters are ordinary "I" and which are CAPITAL LETTER DOTLESS I.
Better to accept the compromise and get on with moving to Unicode. Moreover,
there is a strong doubt that users will "get it right" in future either when
they enter new characters.
[John Cowan]
Q: Why is there no upper-case SHARP S (?)?
There are 139 lower-case letters in Unicode 2.1 that have no direct uppercase
equivalent. Should there be introduced new bogus characters for all of them, so
that when you see an "fl" ligature you can uppercase it to "FL" without
expanding anything? Of course not.
Note that case conversion is inherently language-sensitive, notably in the case
of IPA, which needs to be left strictly alone even when embedded in another
language which is being case converted. The best you can get is an approximate
fit.
[John Cowan]
Q: Is all of the Unicode case mapping information in UnicodeData.txt?
No. The UnicodeData.txt file includes all of the 1:1 case mappings, but doesn't
include 1:many mappings such as the one needed for uppercasing ?. Since many
parsers now expect this file to have at most single characters in the case
mapping fields, an additional file (SpecialCasing.txt) was added to provide the
1:many mappings.
--------------------------------------------------------------------------------
Encoding Forms
Q: Can Unicode text be represented in more than one way?
Yes, there are several possible representations of Unicode data. Chief among
them is UTF-8, discussed in Section 2.3 and Appendix C.3. In addition, there
are compression transformations such as the one described in the Unicode
Technical Report #6: A Standard Compression Scheme for Unicode.
Q: What is a UTF?
A Unicode transformation format (UTF) is an algorithmic mapping from every
Unicode scalar value to a unique byte sequence. Examples are UTF-8 and UTF-16.
(The SCSU compression method is not a UTF because the same string can map to
very many different byte sequences, depending on the capabilities of the
compressor.)
Since every Unicode coded character sequence maps to a unique sequence of bytes
in a given UTF, a reverse mapping can be derived. Thus every UTF supports
lossless round tripping: mapping from any Unicode coded character sequence S to
a sequence of bytes and back will produce S again. To ensure round tripping, a
UTF mapping must also map the 16-bit values that are not valid Unicode values
to unique byte sequences. These invalid 16-bit values are FFFE, FFFF, and
unpaired surrogates.
Q: But what about the byte sequences that are not generated by a UTF. How
should I interpret them?
Let's start with a couple of definitions:
D1. For a given UTF, a byte sequence that cannot be produced from any sequence
of 16-byte values is called an ill-formed byte sequence.
For a given UTF, there are two different types of ill-formed byte sequences.
The first cannot be interpreted by the UTF:
D2. For a given UTF, a byte sequence that cannot be mapped back to a sequence
of 16-byte values is called an illegal byte sequence.
For example, in UTF-8 every byte of the form 110xxxxx2 must be followed with a
byte of the form 10xxxxxx2. A sequence such as <110xxxxx2 0xxxxxxx2> is illegal,
and must never be generated. When faced with this illegal byte sequence while
transforming or interpreting, a UTF-8 conformant process must treat the first
byte 110xxxxx2 as an illegal termination error: for example, either signaling
an error, filtering the byte out, or representing the byte with a marker such
as FFFD (REPLACEMENT CHARACTER). In the latter two cases, it will continue
processing at the second byte 0xxxxxxx2.
The second type of ill-formed sequence can be interpreted by the UTF:
D3. For a given UTF, an ill-formed byte sequence that is not illegal is called
an irregular byte sequence.
To make implementations simpler and faster, transformation formats can allow
irregular byte sequences without requiring error handling. For example, UTF-8
allows non-shortest byte sequences to be interpreted. A UTF-8 conformant
process may map the byte sequence <C0 80> to the Unicode value <0000>, even
though a UTF-8 conformant process must never generate that byte sequence--it
must generate the byte sequence <00> instead. Similarly, it may map the
sequence <ED A0 BF ED B0 80> to the Unicode values <D800 DC00>, even though it
must never generate it--it must generate the byte sequence <F0 90 80 80>
instead.
However, a conformant process cannot use irregular byte sequences to encode
out-of-band information.
Q: You talked about the 16-bit invalid values. Are there any paired surrogates
that are invalid?
Of course, there are no assigned surrogate pairs yet, except for private use
characters, so this topic is not too important yet--you should never be
generating any unassigned characters. However, there are surrogate pairs whose
scalar values are defined as invalid by ISO 10646 (although not for any
particularly good reason). These have the following form:
UTF-16 UTF-8 UCS-4
D83F DFF* F0 9F BF B* 0001FFF*
D87F DFF* F0 AF BF B* 0002FFF*
D8BF DFF* F0 BF BF B* 0003FFF*
D8FF DFF* F1 8F BF B* 0004FFF*
D93F DFF* F1 9F BF B* 0005FFF*
D97F DFF* F1 AF BF B* 0006FFF*
...
DBBF DFF* F3 BF BF B* 000FFFF*
DBFF DFF* F4 8F BF B* 0010FFF*
* = E or F
--------------------------------------------------------------------------------
UTF-16 and UCS-4
Q: What is UTF-16?
Unicode was originally designed as a pure 16-bit encoding, aimed at
representing all modern scripts. (Ancient scripts were to be represented with
private-use characters.) Over time, and especially after the addition of over
14,500 composite characters for compatibility with legacy sets, it became clear
that 16-bits were not sufficient for the user community. Out of this arose
UTF-16.
UTF-16 allows access to 63K characters as single Unicode 16-bit units. It can
access an additional 1M characters by a mechanism known as surrogate pairs. Two
ranges of Unicode code values are reserved for the high (first) and low (second)
values of these pairs. Highs are from 0xD800 to 0xDBFF, and lows from 0xDC00
to 0xDFFF. In Unicode 2.1 and Unicode 3.0, there are no assigned surrogate
pairs. Since the most common characters have already been encoded in the first
64K values, the characters requiring surrogate pairs will be relatively rare
(see below).
Q: Since the surrogate pairs will be rare, does that mean I can dispense with
them?
Just because the characters are rare does not mean that they should be
neglected. It will be important to support surrogate pairs in the future; they
will start appearing in versions of Unicode post-3.0. The fact that the
characters are rare can be taken into account when optimizing code and storage,
however.
Q: Does UTF-16 have an alternative representation?
Yes, all characters represented in UTF-16, both those represented with 16 bits
and those with a surrogate pair, can be represented as a single 4-byte unit in
ISO 10646. This single 4-byte unit corresponds to the Unicode scalar value,
which is the abstract number associated with a Unicode character. The 4-byte
representation of a character is called UCS-4 in ISO 10646.
Q: Should I use UCS-4 for storing Unicode strings in memory?
This depends. We don't anticipate a general switch to UCS-4 storage for a long
time (if ever). It was difficult enough to sell the average US or European
programmer on the idea of using twice as much storage for their characters!
The chief selling point for Unicode was providing a representation for all the
world's characters, eliminating the need for juggling multiple character sets
(and associated data corruption problems). These features were enough to swing
industry to the side of using Unicode (UTF-16). While a UCS-4 representation
does make the programming model somewhat simpler, that is probably not
compelling enough in the foreseeable future to get industry to take the
additional storage hit in switching to internal UCS-4 representations.
Q: How about using UCS-4 interfaces in my APIs?
Given an internal UTF-16 storage, you can, of course, still index into text
using UCS-4 indices. However, while converting from a UCS-4 index to a UTF-16
index or vice versa is fairly straightforward, it does involve a scan through
the 16-bit units up to the index point. In a test run, for example, accessing
UTF-16 storage as UCS-4 characters results in a 10X degradation. Of course, the
precise differences will depend on the compiler, and there are some interesting
optimizations that can be performed, but it will always be slower on average.
This kind of performance hit is unacceptable in many environments.
Most Unicode APIs are using UTF-16. The low-level character indexing are at the
common storage level, with higher-level mechanisms for graphemes or words
specifying their boundaries in terms of the storage units. This provides
efficiency at the low levels, and the required functionality at the high levels.
Convenience APIs can be produced that take parameters in UCS-4 methods for
common utilities: e.g. converting UCS-4 indices back and forth, accessing
character properties, etc. Outside of indexing, differences between UCS-4 and
UTF-16 are not as important. For most other APIs outside of indexing,
characters values cannot really be considered outside of their context--not
when you are writing internationalized code. For such operations as display,
input, collation, editing, and even upper and lowercasing, characters need to
be considered in the context of a string. That means that in any event you end
up looking at more than one character. In our experience, the incremental cost
of doing surrogates is pretty small.
Q: Why are some people opposed to UTF-16?
East Asians (Chinese, Japanese, and Koreans) are understandably nervous about
UTF-16, which sometimes requires two code units to represent a single
character. They have are well acquainted with the problems that variable-width
codes (such as SJIS) have caused. However, there are some important differences
between the mechanisms:
Overlap
In SJIS, there is overlap between the high unit values and the low unit values,
and between the low unit values and the single unit values. This causes a
number of problems:
It causes false matches. For example, searching for an "a" may match against
the second unit of a Japanese character.
It prevents efficient random access. To know whether you are on a character
boundary, you have to search backwards to find a known boundary.
It makes the text extremely fragile. If a unit is dropped from a high-low pair,
many following characters can be corrupted.
In UTF-16, high, low, and single units are all completely disjoint. None of the
above problems occur.
Frequency
The vast majority of SJIS characters require 2 units.
With UTF-16, relatively few characters require 2 units. The vast majority of
characters in common use are single code units. Even in East Asian text, the
incidence of surrogate pairs should be well less than 1% of all text storage on
average. (Certain documents, of course, may have a higher incidence of
surrogate pairs, just as phthisique is an fairly infrequent word in English,
but may occur quite often in a particular scholarly text.)
Q: When will most implementations of Unicode support surrogates?
Since only private use characters are encoded as surrogates now, there is no
market pressure for implementation yet. It will probably be a couple of years
before surrogates are fully supported by a variety of platforms.
Q: Can applications simply use unassigned characters as they wish?
No! No conformant Unicode implementation can use the unencoded values outside
of the private use area.
Only the values in the private use areas (E000..F8FF, F0000..FFFFD, and
100000..10FFFD) are legal for private assignment. However, this is over 137,000
code points, which should be more than ample for the vast majority of
implementations. [F0000..FFFFD and 100000..10FFFD are represented by surrogate
pairs with private-use high surrogates (DB80..DBFF).]
Q: Will UTF-16 ever be extended to more than a million characters?
As stated, the goal of Unicode is not to encode glyphs, but characters. Over a
million possible codes is far more than enough for this goal. Unicode is *not*
designed to encode arbitrary data. If you wanted, for example, to give each
"instance of a character on paper throughout history" its own code, you might
need trillions or quadrillions of such codes; noble as this effort might be,
you would not use Unicode for such an encoding. No proposed extensions of
UTF-16 to more than 2 surrogates has a chance of being accepted into the
Unicode Standard or ISO/IEC 10646.
Q: What about special-purpose implementations that need many code points?
For a particular implementation, if someone really, really wanted a
representation that encoded more characters in a series of 16-bit code units
then a series of private-use characters would work. For example, suppose you
use a representation that consisted one BMP private-use character followed by
one private-use surrogate pair (e.g. three 16-bit units). With such a
representation, you can encode 6400 x 131,072 ( = 838,860,800) private use code
points.
Q: Could you provide some sample code for handling surrogates?
Sure, see Sample Indexing Code
--------------------------------------------------------------------------------
BOM
Q: What is a BOM?
The special characteristics of U+FEFF ZERO WIDTH NON-BREAKING SPACE have been
provided for use by higher level protocols as a signature at the beginning of
certain data streams (primarily unmarked plaintext files). (When used as a
signature, it is referred to as the BOM character, for Byte Order Mark.) Under
those protocols, the BOM may be mandatory in those Unicode data streams.
In the absence of such protocols and when not at the beginning of a text stream,
U+FEFF is given its normal interpretation, as ZERO WIDTH NON-BREAKING SPACE,
and is part of the content of the file or string.
Q: Where is a BOM useful?
Where a BOM is useful is with files that are typed as text, but not known to be
in either big or little endian format--it also serves in that situation to be a
hint that the text is Unicode, and not a legacy encoding.
Q: I am using a protocol that has BOM at the start of text. How do I represent
an initial ZWNBSP?
Use the sequence FEFF FEFF.
Q: When a BOM is used, is it only in 16-bit Unicode text?
No, a BOM can be used as a signature no matter how the Unicode text is
transformed: UTF-16, UTF-8, UTF-7, etc. The exact bytes comprising the BOM will
be whatever the Unicode character FEFF is converted into by that transformation
format. In that form, the BOM serves to indicate both that it is a Unicode file,
and which of the formats it is in. Examples:
Bytes Encoding Form
00 00 FE FF UCS-4, big-endian
FF FE 00 00 UCS-4, little-endian
FE FF UTF-16, big-endian
FF FE UTF-16, little-endian
EF BB BF UTF-8
Q: How do I tag data that does not interpret FEFF as a BOM?
Use the tag UTF-16BE to indicate big-endian UTF-16 text, and UTF-16LE to
indicate little-endian UTF-16 text. If you do use a BOM, tag the text as simply
UTF-16.
Q: Why wouldn't I always use a protocol that requires a BOM?
Where the data is typed, such as a field in a database, a BOM is unnecessary.
For example, if a text data stream is marked as UTF-16BE, it is unnecessary to
have a BOM. In fact, it is very undesirable to tag every string in a database
or set of fields with a BOM, since if the strings are concatenated without
removing the BOM you get into trouble. Moreover, it also means two data fields
may have precisely the same content, but not be binary-equal (where one is
prefaced by a BOM).
Q: Can you summarize how I should deal with BOMs?
Here are some guidelines to follow:
A particular protocol (e.g. Microsoft conventions for .txt files) may require
use of the BOM on certain Unicode data streams, such as files. When you need to
conform to such a protocol, use a BOM.
Some protocols allow optional BOMs in the case of untagged text. In those cases,
Where a text data stream is known to be plain text, but of unknown encoding,
BOM can be used as a signature. If there is no BOM, the encoding could be
anything.
Where a text data stream is known to be plain Unicode text (but not which
endian), then BOM can be used as a signature. If there is no BOM, the text
should be interpreted as big-endian.
Where the precise type of the data stream is known (e.g. Unicode big-endian or
Unicode little-endian), the BOM should not be used.
--------------------------------------------------------------------------------
Unicode and ISO 10646
Q: What is the relation between ISO-10646 and Unicode?
In 1991, the ISO Working Group responsible for ISO/IEC 10646 (JTC 1/SC 2/WG 2)
and the Unicode Consortium decided to create one universal standard for coding
multilingual text. Since then, the ISO 10646 Working Group (SC 2/WG 2) and the
Unicode Consortium have worked together very closely to extend the standard and
to keep their respective versions synchronized.
Unicode 2.1 has the same character repertoire as ISO/IEC 10646-1:1993, and and
already incorporates all the Amendments through Amendment 7 to 10646-1, plus
any relevant information from Technical Corrigenda #1 and #2 to 10646-1.
[Ed Hart, Ken Whistler]
Q: How do I get a copy of ISO 10646 that is equivalent to Unicode 2.1?
You will need IS 10646-1:1993, plus Amendments 1-7 and applicable corrigenda.
However, although the character codes are synchronized between Unicode and
ISO/IEC 10646, the Unicode Standard imposes additional constraints on
implementations to ensure that they treat characters uniformly across platforms
and applications. To this end, it supplies an extensive set of functional
character specifications, character data, algorithms and substantial background
material that is not in ISO/IEC 10646.
You can order direct from ISO Central Secretariat (at
http://www.iso.ch/cate/d18741.html) for a total of 648,50 CHF, or in the U.S.
from ANSI (at http://web.ansi.org/public/catalog/cat_top.html) for a total of
585 USD for the standard plus Amendments 1-7. Then download Technical
Corrigenda #1 and #2 off the SC2 website by contacting the SC2/WG2 convener at
the address provided via http://www.iso.ch/meme/JTC1SC2.html.
[Ed Hart, Ken Whistler]
--------------------------------------------------------------------------------
Copyright
Copyright ? 1998-1999 Unicode, Inc.. All Rights Reserved. The Unicode
Consortium makes no expressed or implied warranty of any kind, and assumes no
liability for errors or omissions. No liability is assumed for incidental and
consequential damages in connection with or arising out of the use of the
information or programs contained or accompanying this technical report.
Unicode and the Unicode logo are trademarks of Unicode, Inc., and are
registered in some jurisdictions.
--
☆ 来源:．BBS 荔园晨风站 bbs.szu.edu.cn．[FROM: bbs@210.39.3.80]
[回到开始]
[上一篇][下一篇]
荔园在线首页 友情链接：深圳大学 深大招生 荔园晨风BBS S-Term软件 网络书店