This report was produced by a CEN/TC 304 Project Team, set up in June, 1998, as one of several to carry out the funded work program of TC 304 (documented in CEN/TC 304 N 666 R2). A first draft was discussed at the TC meeting in Brussels in November, 1998. This revised draft is circulated for comments within the TC. A final draft will be presented for approval at the next TC plenary meeting (April, 1999). The approved version will then be sent to the CEN BT for approval.

There exist today a large number of standards and related specifications concerning character repertoires and their coding in the form of official as well as manufacturer standards and intended for a wide range of applications and uses. Furthermore, there are character set standards for data communication and there are standards developed specifically for telecommunications applications. The situation can be very confusing to the non-expert user and to people involved in procurement.

The user of IT systems normally does not have to concern himself with these types of standards. However, there may be situations where he has to be able to express his needs for certain character repertoires necessary for his work; it may also happen that he, when involved in work together with other parties using other systems, needs to be able to interpret other people's specifications given in the form of reference to standards.

The procurer of IT systems should be able to specify his requirements in the form of reference to established standards.

A particular purpose of the report is to give guidance for public procurement in Europe. Since there is an EC directive and a council decision for such procurement requiring the use of official European standards above certain procurement amounts, the report concentrates on such standards. There may be future editions, in which case more attention will be given other types of standards. (See also section 7.)

The main purpose of this report is to give guidance to users and procurers by explaining the purposes and relationships of the official standards in the domain of data communication. Explicit guidance is given in paragraphs marked with 4 .

The text is presented on two levels. The first level, contained in the body of the report, provides a general coverage of character repertoires, coding and uses. The second level, contained in the two annexes, provides much more detailed, tutorial information. The reader who finds the level of technical detail to deep may be better served by the "Manual: Standards for the electronic interchange of personal data: Part 5 - Character sets" (see References).

Further information on character sets and their standardization can be found in the document "Language automation world-wide: The development of character set standards" and on the Letter Database web site (see References).

The main body of this report is aimed primarily at the non-technical person who needs to become familiar with use of character set standards in Europe for various purposes in an IT environment. This audience will include managers/decision makers and their advisors; administrators (for procurement purposes); technicians (for programming and system development purposes); standardisers; perhaps also journalists.

The concepts of characters and their coding is introduced in section 5, and a conceptual model on the use of coded character sets is provided in section 6. The guide concentrates on official character set standards. However, there is a range of other standards for character sets that are not official, and there are also specifications concerning associated topics such as rules for ordering character strings. Section 7 goes on to place the official standards in the wider context of these other standards. Sections 8 and 9 describe a range of official character set standards with an international and a European scope respectively. Section 10 introduces a number of procurement issues, and section 11 provides sample text that may be used as the basis for inclusion in (public) procurement specifications for IT systems and software.

In addition, the guide has two annexes which contain a much more technical description of official character set standards.

The activities of CEN/TC304, the committee responsible for the promulgation of character set and related specifications in Europe, are described in section 12, and finally pointers for further reading and research are given in section 13.

The technical scope of this guide is primarily limited to official character set standards promulgated by ISO/IEC and CEN, as opposed to official telecommunications standards and manufacturer standards. However, an overview of all types of standards is given in section 7. The guide furthermore concentrates on European issues; thus character set standards for non-European languages are not covered.

The guide is mainly intended as an introduction for people who need to familiarise themselves with the concept of character sets and their coding; e.g. managers/decision makers and their advisors; administrators (for procurement purposes); technicians (for programming and system development purposes); standardisers; perhaps also journalists. Particular emphasis is placed on its use by procurers.

The following terms are used in the body of this report and the official definitions are given here where they exist. They are taken from the standards ISO/IEC 9541:1991 and ISO/IEC 10646-1:1993, except when denoted by an *.

(character) repertoire: A specified set of characters that are each represented in a coded character set.

control function: An action that affects the recording, processing, transmission, or interpretation of data, and that has a coded representation containing one or more bit combinations.

*Note - A bit combination in this context is a 7- or (more commonly) 8-bit byte.

control character: A control function the coded representation of which consists of a single bit combination.

*Note - A control character is not strictly spoken a "character" but is called that way because its coded representation is of the same type as that of a coded graphical character.

coded character set (character set): A set of unambiguous rules that establishes a character set and the one-to-one relationship between the characters of the set and their coded representation.

*code table: A tabular representation of a coded character set, showing also the coded representations.

*code page: Synonym for code table, used in the IBM environment.

*code space: The numeric domain occupied by all bit combinations used for the coding of a coded character set.

transliteration: The process which consists of representing the characters of an alphabetical or syllable writing system by the characters of a conversion alphabet.

Note - In principle, a transliteration should be a one-to-one conversion.

*fall-back: A non-reversible transformation consisting of the substitution of an output character which cannot be represented on the output device by one or more characters which can.

combining character: A member of an identified subset of a coded character set, intended for combination with the preceding or following graphic character, or with a sequence of combining characters preceded or followed by a non-combining character

*diacritic, diacritic mark: A mark intended for the association with a letter (e.g. acute accent).

glyph: A recognisable abstract graphic symbol which is independent of any specific design.

For the presentation of written text we use letters, digits and punctuation marks. Often we also use special symbols such as currency signs. All of these are called characters, and the collection of characters for a specific purpose, such as the presentation of text in a specific language, is called in the standardisation context a character repertoire. The most common type of repertoire is of course the alphabet of a language, complemented by the ten digits and a set of special characters.

Although the glyph concept is important for the definition of character repertoires, it is not central to the theme of this guide. The reader who wishes to obtain more information about glyphs is referred to ISO/IEC TR 15285, An operational model for characters and glyphs (see References).

Almost every language has its own character repertoire. However, the fact that many European languages have a large number of characters in common naturally facilitates the work on defining character repertoires for Europe. In CEN/TC 304 there is a separate activity on providing a catalogue of the alphabets of indigenous languages, information on which can be found at http://www.stri.is/tc304/......

In IT systems a character is represented by a 7- or 8-bit combination, usually expressed as a numeric code. A character repertoire with its corresponding set of codes is called a coded character set or just character set. Such a set is often represented graphically in the form of a code table (Figure 1), which also illustrates the principles of the distribution of the codes, the code structure. Furthermore, the totality of the bit combinations used for a coded character set is called its code space.

Coded character sets are used for different purposes in computer systems, and the code structures may therefore vary. For instance, a coded character set used for interchange purposes often needs codes to be reserved for control characters, so that these may be included in the interchange data stream. However, a coded character set used for processing purposes may not need such reserved areas, which instead are often only used to represent more graphic characters. Examples of the latter are the manufacturer standards known as PC code pages.

Early IT systems had severe size limitations. Therefore, the character codes had to be kept small. The earliest codes occupied 5 and 6 bits; later 7 and 8 bits have been used. These provide a coding capacity for 32, 64, 128 and 256 characters respectively. However, even with an 8-bit representation it is not possible to support all European languages in a single coded character set. Thus coded characters sets proliferated. As long as an application (and the character set it used) was restricted in use to a single country or geographical region, this proliferation did not create problems, since a character set could be chosen to support the limited number of languages for that region. However, due to the requirements of international trade and the increase in travel, the limitation in the number of characters in one coded set has caused great problems of application interoperability.

In order to avoid a very large number of private character set specifications, many with overlapping scope and leading to interoperability problems, standardisation was needed. It was carried out both by the official standardization organisations and by the manufacturers, most notably by IBM, Apple and Microsoft.

Modern IT systems no longer have the earlier restrictions in size, and a solution is now available which uses a code space sufficient to accommodate the characters of every language in the world in one and the same coded character set. However, since the old solutions seem likely to continue to exist until perhaps 2025, the old problem may remain acute for some time. There will be the added complication of using the old and the new systems together as well as how to migrate, in an orderly fashion, to the new system.

For IT processing purposes, it is necessary to indicate within a data stream where some action is required, e.g. a carriage return or new line. Such actions are performed through control functions, which do not have graphical representations. Over 160 control functions have been standardized. Some of them, such as the carriage return, are represented by a single control character, which, even though it does not have a graphical representation, has a coded representation of that type and can therefore be included in a code table. Others are represented by a sequence of characters with a special introducing control character at the beginning of the sequence.

Figure 1 below illustrates the character handling model. It represents a simplified IT scenario which consists of two computer systems connected by a communications link. The purpose is to show the different aspects of the handling of characters by users and computer systems and thus introduce basic concepts that will be used in the following sections of the report. It is also intended to help differentiate between the roles of the user(s) and the procurer in the context of this guide.

The input function provides for the entering of data into a computer system. Figure 2 uses a keyboard for input, but any device capable of entering character data may be used.

• For the user, the main issue is whether or not there is available in the computer system a character repertoire for input which is sufficient for his needs. For the procurer, the main issue is to produce a procurement specification which satisfies the input needs of all intended users of the product.

Note that the representation of the input text on the monitor screen is a result of both the processing function, e.g. a word processor, and an output function (to the screen).

The main keyboard standard is ISO/IEC 9995, Keyboard layouts for text and office systems. National keyboard standards have, in general, been promulgated based upon this international standard. Keyboard standards are related to character set standards but are not central to the theme of this guide. CEN/TC304 has a separate activity on European keyboard standardisation, information on which may be found at http://www.stri.is/tc304/......

The processing function provides for the manipulation of data according to the needs of an application.

Once input, the data is expressed in some internal computer system code. In addition, other information may be associated with each character such as colour, emphasis level and font. Such information is usually intended for some document processing function. Thus the system internal code structure may be quite complex. However, at its heart is the character code itself; document handling and processing is outside the scope of this guide.

Most commercially available computer systems do not use standardised character sets for internal representation of character data, but proprietary character sets or manufacturer specifications.

• The user needs to be able to have all input characters processed, while, again, the procurer needs to produce the appropriate procurement specification.

A particularly common requirement on the processing function is that it be able to order character based data. The main ordering standard is ISO/IEC 14651, International string ordering - Method for comparing character strings and description of a default tailorable ordering. Standards for ordering, while related to character set standards, are not central to the theme of this guide. In CEN/TC 304, there is a separate activity on European standardisation of ordering, information on which may be found at http://www.stri.is/tc304/......

The interchange function provides for the interchange of data between computer systems. Since the character sets for processing are generally defined by manufacturer specifications, they are likely to be different in two different computer systems between which data are to be exchanged. Thus a character set for interchange is needed which will have to be different from one or both of the processing character sets. This is where character set standards become very relevant to reduce the number of interchange character sets - potentially one for every possible pair combination of different computer systems.

The main problem here is that there may not be a one-to-one correspondence between the characters in the character set for processing and those in the character set for interchange. In such cases less than trivial transformation functions are required at the interfaces between processing and interchange; see clause 10.2. Such transformations may incur a loss of information.

• The user requirements will relate to the functionality of the interchange (e.g. what loss of information, if any, may be accepted). Such requirements may also include policy decisions of a more technical nature, e.g. as to what code structure to use. As before, the procurerâs role is to transform the requirements into technical specifications.

The output function is the process of converting the internal coded representations of the characters to a visual representation on a display or hard copy device. The output character sets may be different depending of the output medium.

The handling of output to physical devices is usually an internal computer system function. Application programs, such as word processing packages, normally have the ability to control also the rendition of the output. This includes the use of fonts, both type and size, and also the use of various levels of emphasis and colour. In some cases, information which specifies particular values of these attributes is carried with the individual character codes right from the time of input. As already stated, these features are outside the scope of this guide.

The main problem is when the output character set is smaller than the character set for processing. The computer system software has to substitute one or more characters for those which cannot be represented by the output function (see clause 10.2). Again, such transformations may incur a loss of information.

• The user requirements will relate to the functionality of the output (e.g. what loss of information, if any, may be accepted). As before, the procurer's role is to transform the requirements into technical specifications.

Each country or region in Europe (and elsewhere) has cultural conventions which affect the manner in which character sets are used by application programs. Such conventions include, but are not restricted to, ordering (already mentioned), numeric formatting, monetary formatting, date and time conventions, affirmative and negative answers, the use of special characters and personal name rules.

Cultural conventions primarily affect the processing function but may also impact the other functions. They are related to the use of character set standards but are not central to the theme of this guide. In CEN/TC304 there is a separate activity on cultural conventions which is developing and maintaining a registry of such conventions. Information may be found at http://www.stri.is/tc304/......

There are four main categories of character set standards:

• Official standards for the computer and data communications industry, promulgated by international standards organisations such as ISO/IEC and CEN, but also by national standards organisations. They are primarily intended for input (mainly for the specification of character repertoires) and for the interchange function in general applications.

• Official standards for the telecommunications industry, promulgated by ITU-T (formerly CCITT; the standards are called Recommendations) and ETSI, mainly intended for the interchange function in telematics applications.

• Manufacturer standards for the computer and data communications industry, developed by industry groupings such as the UNICODE consortium and individual companies such as IBM and Microsoft. They are mainly used for the input and processing functions but also for the interchange function in internal (proprietary) networks.

• Related standards, which involve character sets such as keyboard standards, ordering standards and transformation standards.

As already explained, this guide concentrates on the first of these categories (but does not cover national standards). In the remainder of this section further information is given on the other categories.

Character set standards promulgated by ITU-T and by ETSI support applications primarily defined and promoted by the telecommunications companies. Thus for Teletex, Recommendation T.61 was developed. It was subsequently replaced by T.51, which is also used as the basis for character set support in other telematic applications such as the X.400 Message Handling Service and the X.500 Directory Service. There are some links, however, with ISO/IEC standards. T.51 is equivalent to ISO/ IEC 6937 (see section 8); and there is also T.50, equivalent to ISO/IEC 646 (see section 8). This alignment is due to collaboration between the two standards organisations. Nevertheless, ISO/IEC 6937 is used primarily for telematic applications and services.

ETSI has developed character set standards in support of radio paging (ERMES), GSM and RDS, but these standards are not aligned at all with ISO/IEC standards.

Manufacturer standards can be grouped into IBM EBCDIC variants and those in support of personal computers such as the PC and the Macintosh. All use 8-bit codes. The EBCDIC code pages have their own defined structure together with an invariant set of characters and code positions. This is similar in concept to the parts of ISO/IEC 8859 (see section 8). Indeed some EBCDIC code pages have the same repertoire as parts of ISO/IEC 8859, which makes it straightforward to perform mappings between the two environments. Other EBCDIC code pages are designed to support specific countries or regions. More information on this may be found in the IBM document "Character Data Representation Architecture" (see References).

Both IBM and Microsoft have designed PC code pages, and each page is aimed at supporting specific countries or regions. PC code pages do not in general comply with the 8-bit code structure specified in ISO/IEC 2022 (see section 8), and mappings between the two environments may result in the loss of information.

IBM EBCDIC and PC code pages are described in Appendix I of "IBM National Language Support Reference Manual Volume 2" (see References).

The document "Comparisons of Standardized Character Sets for Europe" provides mappings of the differences between various official and manufacturer standards (see References).

In addition to standards on the definition of character repertoires and coding, there are standards relating to the handling of character sets and other character set issues.

Keyboard standards are concerned with the input to computer systems of character information. They cover such topics as keyboard layouts. There is one international standard plus a range of national standards, most of which are based on the international standard.

Ordering standards are concerned with sorting character strings or sets of strings into some order. There is an international standard, and work is under way in Europe on the ordering of European repertoires. A related activity concerns the matching of character strings, for use in, for instance, search engines.

Transformation standards are concerned with rules for mapping the characters from one repertoire onto another, and from one code system onto another. They are sometimes needed when character information from one computer system needs to be transferred into a different computer system due to a mismatch of functionality. An example is the mapping of an official standardised character set onto EBCDIC, so that character based information may be processed in an IBM machine. There are a number of different types of transformation needed for various scenarios. See also clause 10.3.

This section describes internationally standardised character sets (i.e. both repertoires and coding) used primarily for input (mainly for the specification of repertoires) and interchange. For processing, most IT systems use proprietary manufacturer standards.

Because of the need to combine different standards and also in order to make the standardisation itself more consistent and effective, a common platform standardising the principles for code structure, code extension, implementation and registration has been established. The standards which define that platform may be called framework standards.

The international character set standards in this area may be classified in the following way:

• 7- and 8-bit framework standards
• 7- and 8-bit character set standards
• The universal character set (UCS) standard (multiple octet)
• The control function standard

See also Figure 3 below.

• ISO/IEC 2022:1994, Information technology - Character code structure and extension techniques, is the basic framework standard. It provides the framework for the definition of all the 7- and 8-bit coded character sets and their use. In particular, it defines a complex code extension facility which allows considerably more characters to be used in a specific environment than would otherwise be the case with a single 7- or 8-bit code space. It also partitions the 7- and 8-bit code spaces between the representation of graphic characters and control characters. See also Annex A.

• ISO/IEC 4873:1991, Information technology - ISO 8-bit code for information interchange - Structure and rules for implementation, is a refinement of ISO/IEC 2022 which concentrates specifically on the implementation and use of the 8-bit coding. It lays the formal basis for the specification of two 8-bit character set standards - the various parts of ISO/IEC 8859 and ISO/IEC 10367 (see section 9).

• ISO/IEC ISP 12070:1996, Information technology - International Standardised Profiles FCSnnn - Character set 8-bit code structure based on ISO/IEC 2022 - Part 1: FCS111 - 2022 Option 1, is a profile of both ISO/IEC 2022:1994 and ISO/IEC 4873:1991. It goes into considerable detail concerning the use of character sets within Open System Interconnection (OSI) protocols and, in particular, concentrates on conformance requirements (in the interests of interoperability). It is particularly the latter which makes this standard relevant in this context, whereas its OSI aspects are less interesting nowadays.

Note - A profile is a selection of options from one or more standards, constituting a more narrow form of specification than the base standard(s) it refers to.

• ISO 2375:1985, Data processing - Procedure for the registration of escape sequences, governs the registration of coded character sets. In the ISO/IEC 2022:1994 framework, the extension mechanisms use escape sequences to control the use of more than one registered coded character set. Such an escape sequence consists of the Escape control character followed by a variable number of octets. In this way, the sending implementation in an interchange can inform the receiving implementation about the elements of the code structure that will be used in the interchange.

A registration authority allocates values for the escape sequences to be used with the registered coded character set. The registration authority is the Information Processing Society of Japan/Information Technology Standards Commission of Japan (IPSJ/ITSCJ), and the register may be accessed on-line at http://www.itscj. ipsj.or.jp/ISO-IR/. All standardised coded character sets are registered, but not all registered coded character sets are standardised. There are currently over 200 registrations. A coded character set may be identified in text, such as a procurement specification, by the sequence ISO-IR nnn, where nnn is the registration number.

• Most users and procurers need not concern themselves overmuch with ISO/IEC 2022:1994 or ISO/IEC 4873:1991, since their provisions are usually called up by the referring standards. For this reason, they need not be referred to directly in procurement specifications. However, the sophisticated user with a complex requirement, who may be mixing and matching the use of registered character sets rather than standardised character sets, will probably need to become familiar with them. In this case, the procurer will also need to make reference ISO/IEC ISP 12070-1 in order to ensure the necessary conformance. This ISP is also relevant in procurement situations which include the proposed use of products using OSI protocols.

• Users and procurers are advised not to use national variants of ISO/IEC 646:1991 .

The set of characters which only includes the non-optional characters is called the Invariant Set. There are default allocations of characters to the optional code positions, and when those are used, the character set is called the International Reference Version (IRV). The IRV is registered as ISO-IR 6. It was changed in the 1991 edition of the standard with the currency sign being replaced by the dollar sign.

The standard also defines a default set of control characters to be used where applicable. The IRV:1991 together with the default control character set is identical to ASCII (American Standard Code for Information Interchange), a name in common usage in the industry. This character set does not contain any letters with accents or diacritical marks. It is therefore unsuitable for use in representing most European languages.

• ISO/IEC 8859:nnnn, Information technology - 8 bit single-byte coded graphic character set, is a multi-part standard whose parts have been promulgated at different times. Each part defines an 8-bit character set with a specific language coverage. Although language coverage has been the driving force behind the proliferation of parts, each part is designed for widespread general use such as normal commercial and administrative applications. They have not been designed for specialist minority applications.

Each part of ISO/IEC 8859 contains a code table made up of two halves (Figure 4). The left half is identical to the graphic characters of the IRV of ISO/IEC 646:1991 coded in 8 bits. The right half is specific to the part in question. Together, the two halves define a self contained 8-bit coded character set of up to 191 characters. There are currently 15 parts to ISO/IEC 8859, falling into two categories. In the first category, containing 9 parts, the second table contains extra Latin characters. The title of each of these parts is Latin Alphabet No. n where n lies in the range 1 to 9. In the second category, the second table contains characters from a non-Latin script. The title of each of these parts is Latin/xxx where xxx is the name of the second script (e.g. Latin/Greek). Each part of ISO/IEC 8859 supports a range of languages. They are listed in Annex A in the section on ISO/IEC 8859.

Note - In products supporting ISO/IEC 8859, no code extension is used. However, since the right halves of the code tables - the ones specific to each part of the standard - are all registered, any one of them can form part of larger coded character sets with the use of code extension techniques. For such implementations, reference must be made to the registration number of the code table and not to the corresponding part of the standard.

• ISO/IEC 6937:1994, Information technology - Coded graphic character set for text communication - Latin alphabet, is a standard primarily used in ITU-T telematic applications. It is equivalent to the ITU-T recommendation T.51. It overcomes the restrictions placed by the 8-bit code space by using two octets for characters with accents or diacritic marks: a non-spacing diacritic mark followed by the character for which the diacritic mark is to apply. The standard specifies a repertoire of 333 characters. However, it has not found favour outside the world of telecommunications. It is therefore not recommended for use by European IT applications.

• ISO/IEC 10367:1991, Information technology - Standardised coded graphic character sets for use in 8-bit codes, attempts to overcome the 191 character limit of the parts of ISO/IEC 8859:nnnn by specifying the code tables of ISO/IEC 8859:nnnn parts for use with the code extension facilities of ISO/IEC 2022:1994, qualified by ISO/IEC 4873:1991. This effectively allows coded character sets of 382 characters to be defined and used. The standard also defines how ISO/IEC 6937:1994 may be used in conjunction with the non-Latin script code tables defined in the right half of the various parts of ISO/IEC 8859:nnnn. A problem with this standard is that it is now out of date, since it refers only to the first 9 parts of ISO/IEC 8859:nnnn whilst there will soon be 15 parts. However, in principle the specification used in ISO/IEC 10367:1991 may of course be applied as well to the remaining parts of ISO/IEC 8859:nnnn.

The merit of this standard is that it specifies a functionality which goes somewhat beyond that of EN 1923 (see clause 9.1) while still using the "old" 8-bit technology.

It is obvious that the limitations of both 7- and 8-bit coding create problems in a world where increasingly texts in very different languages need to be communicated between computer systems. Some years ago, the radical step was taken to create a standard with enough code capacity to allow the coding of all alphabets in the world within one framework.

• ISO/IEC 10646-1:1993, Information technology - Universal Multiple-Octet Coded Character Set (UCS), Part 1: Architecture and Basic Multilingual Plane, is a relatively new standard which aims to overcome all the shortcomings of the 7- and 8-bit character set standards.

ISO/IEC 10646 defines a character set and a code structure of up to 4 octets. The main aim was to provide sufficient code capacity so that the alphabets of all the known languages in the world, together with a large range of special characters, could be accommodated. It has to be said that four octets is an overkill, but this was kept to cater for any possible future requirement. It was decided to concentrate at first on populating the lowest two octets of the code space, and this space is known as the Basic Multilingual Plane (BMP).

Since publication, the standard has been subject to a number of amendments, most of which add further language support to the BMP.

It is planned that the BMP will contain all characters, including combining characters, needed to write all the known living languages of the world, with the exception of some of the Chinese and Japanese ideographs. It should be noted that over 27,000 Han characters for Japanese and Chinese are assigned to the BMP, so there is significant support for the languages of those countries which only require a few thousand characters for the large proportion of applications. The BMP certainly already contains all the characters needed for the vast majority of European languages. For this reason, a two octet as well as a four octet representation is specified for use when the application environment only makes use of the BMP. Other coded representations are specified to cater for situations where more than the BMP is used but where better efficiency than use of the four octet form is needed. More detailed technical information may be found in Annex B.

• The UCS represents the future direction for coded character sets and is being implemented by a wide range of suppliers. For this reason, users and procurers are recommended to consider seriously its use for new systems in preference to 7- and 8-bit coded character sets. It should be noted that, whereas 7- and 8-bit character sets are primarily used for interchange, there is evidence that the UCS will be used for the processing function as well. However, 7- and 8-bit systems will continue to exist for some considerable time, possibly for up to 25 years. If interoperability with such systems is required, it is important for the procurer to make sure that suppliers of new products can provide this before he commits to use or purchase.

A procedure has been agreed for the submission of proposals for the addition of subsets to the base standard. There are several reasons for this. Many systems have quite modest character set requirements, which are application dependent. There are severe restrictions in small implementations such as for mobile telephones, mobile digital radio systems and set top boxes for digital television. Furthermore, there is in Europe a need to identify a character set specifically to support all European languages. Submissions are normally channelled through the national standards organisations.

An equivalent but not identical specification to this standard is also published by an industry consortium called UNICODE. Although the two are separately published, every attempt is made to ensure that they are kept in line. In particular, both publications define the characters contained in the Basic Multilingual Plane (BMP).

The UNICODE standard plays an important role from the point of view of procurement. As already noted there have been a number of amendments to ISO/IEC 10646-1, and over the past few years the appearance of amendments has been continuous. The standard changes every time an amendment is ratified. This means that it is not reasonable for suppliersâ implementations to be kept in synchronisation. The UNICODE standard, on the other hand is not changed very often and so suppliers can keep in synchronisation with it. Version 2 of the UNICODE standard is identical position for position with the first edition of ISO/IEC 10646-1 plus its first 7 amendments. Version 3 of the UNICODE standard, due for publication in 1999, will be position for position identical with the second edition of the international standard, which will incorporate the first 31 amendments. These publications represent significant points for the industry to synchronise its products.

This standard was originally defined for use in the 7- and 8-bit code structures. However, ISO/IEC 10646-1 has provisions which allow control functions to be used in the UCS code structure.

In order to provide specifications particularly suited for European applications, CEN has standardised implementations of ISO/IEC standards, including subsets of the UCS standard. CEN promulgates ENs (European Standards) and CWAs (CEN Workshop Agreements). An EN is promulgated by a full CEN Technical Committee and is ratified by a ballot involving the national standard organisations (NSOs) belonging to CEN. A CWA does not carry the authority of a full EN and is promulgated by a Workshop whose membership is open to all comers. A CWA is not subject to NSO ballot; its authority rests on the fact that it is an agreement - and commitment - based on a wide marketplace consensus.

EN 1923 specifies how these repertoires are coded using the extension mechanisms of ISO/IEC 2022:1994, and how they can be combined. A set of combination options is given. (Not all possible combinations are allowed.)

The CEN Workshop Agreement (CWA) nnn specifies three subsets of ISO/IEC 10646-1:1993. These are the so called Multilingual European Subsets (MES). The purpose of the CWA is to identify subsets of the BMP for specific use in Europe so that they may be added to the UCS standard.

This CWA is at the time of writing (February, 1999) not finally agreed, which means that its publication in that form is not guaranteed. However, the work towards this particular goal will continue.

• MES-1 contains the 333 characters of the repertoire of ISO/IEC 6937:1994 plus Latin capital letter Eth (Icelandic) and the Euro sign. It was created for reasons of compatibility with LL8 and provides a straightforward migration path for older applications. The characters in this subset are primarily intended for use in official languages and several minority languages in a contiguous block of European countries to the west of the Russian Federation, Belarus and the Ukraine, and also in Turkey.

• MES-2 is much larger than MES-1 and is a superset to that repertoire. Its characters are primarily intended for use in official languages and several minority languages in a contiguous block of European countries to the west of the Caspian Sea, excluding Georgia (which uses Georgian script), and Armenia (which uses Armenian script). MES-2 aims to satisfy user requirements in general purpose data and text processing applications in typical office environments in Europe. It specifies a collection of basic Latin, Greek, and Cyrillic letters, together with digits, punctuation and other symbols, broadly similar to that used in EN 1923, in ISO/IEC 10367 (except for Hebrew and Arabic characters in ISO/IEC 10367), and in the various parts of ISO/IEC 8859 which use European scripts. It is also broadly similar to the repertoire used in the "WGL4.0 Character Set" (really a character repertoire) defined in the increasingly used OpenType specification. MES-2 furthermore includes the full range of polytonic Greek characters, used widely in Greece until 1982 and still enjoying usage there, whether in ordinary contexts or in the context of the study of Ancient Greek.

• MES-3 is much larger than MES-2 and is a superset to that repertoire. It includes all the collections of ISO/IEC 10646-1 which contain the characters of the Latin, Greek, Cyrillic, Armenian, and Georgian scripts, together with those collections of symbols used academically, commercially, and scientifically in Europe. By including combining characters and phonetic characters of the Latin alphabet (including the International Phonetic Alphabet, I.P.A.), MES-3 also provides for the needs of transliteration and transcription of many of the worldâs languages into the Latin script, as well as the needs of European publishers (of dictionaries, encyclopaedias, phrase books, and textbooks) who routinely make use of the I.P.A. It also aims to provide all characters used in several more existing European and International character set standards, as well as some proprietary character sets. It provides a European specification for the needs of more specialised groups in government, industry, publishing, academia, and the private sector, than MES-1 and MES-2 do.

MES-3 is defined in two forms: One is a non-fixed subset (MES-3A), which permits automatic inclusion in it of any new characters added to the BMP whose coding positions fall within the collections which define this subset. The other is also the fixed subset (MES-3B), which is invariant over time.

In order to support the move to EMU in the Euro zone of the EU, the Euro sign has now been included in the following of 8-bit code tables.

• ISO/IEC 8859-15:1998 - Latin-9
• ISO-IR 203 - European Supplementary Latin Set ("Latin 9")
• ISO-IR 204 - Supplementary set for Latin-1 alternative with EURO SIGN
• ISO-IR 205 - Supplementary set for Latin-4 alternative with EURO SIGN
• ISO-IR 206 - Supplementary set for Latin-7 alternative with EURO SIGN

In all of these, the Euro sign is in the code position usually occupied by the currency sign. The Euro sign is already included in ISO/IEC 10646-1 by amendment 18 and is also included in all three Multilingual European Subsets (MES).

The procurement issues described here are related to specific character handling functions (see section 6).

The user does not need to be concerned with the coded representation of the input character set since a character, once input, has a code defined by the internal processing. What may be an issue here is the specification of the input character repertoire, which must be able to support the user requirements.

For the same reason, neither does the user need to be concerned with the coded representation of the output character set. Again, the repertoire needs to be sufficient to support his requirements. In particular, there should be font support for that repertoire. Character transformation may be an issue if the product cannot support the full output repertoire needed; see clause 10.2.

The procurement issues will centre around the interchange character set. As in the other cases, the repertoire needs to be sufficient to support the requirements of the user. The main issue will be the code structure. The industry is in a state of transition. The 8-bit code structure of ISO/IEC 2022:1994 has been around for some considerable time and is the safe, but limited, option. The future lies with the multi-octet code structure of ISO/IEC 10646-1:1993, but it will not for some time be available in all receiving systems. Still, this guide recommends that new procurements should specify the multi-octet code structure wherever possible. However, there may be cases where the new system has to operate in an environment which overwhelmingly uses 8-bit coding. If so, that code structure should be used. Increasingly, however, there will be situations where the new system has to operate with both code structures, and then the availability of dual support has to be considered.

If the 8-bit code structure is chosen for the interchange function, this guide recommends that, in Europe, the repertoire(s) are chosen from those identified in EN 1923:1998.

• For Latin script applications, there is a choice of four repertoires. It is unlikely that the IVL or IL repertoires will be sufficient for European operation and a choice of these alone is deprecated, except IL for use in EDIFACT.

• For Western European language application, the BL repertoire should be sufficient.

• If Eastern European languages have to be taken into account, then the LL8 repertoire is more applicable. The Greek (BG) and Cyrillic (BC) repertoires can be added where appropriate, but they cannot be combined with LL8 unless the code extension facilities of ISO/IEC 2022 are used. It should be noted, however, that there may be significant limitations in these repertoires for some minority language operation and for some specialist technical applications. For such cases, it is recommended to seek specialist advice in the choice of coded character sets outside EN 1923:1998.

• If the multi-octet code structure is chosen for the interchange function, this guide recommends that one of the Multilingual European Subsets (MES) be selected for the interchange repertoire. MES-1 is almost identical to LL8. MES-2 provides a much more comprehensive coverage for European languages, whilst MES-3 guarantees support for nearly all indigenous European languages, specifically including minority languages.

The repertoire selection indicated above is a minimum requirement. The supplier may go further.

The choices of repertoires for the four character handling functions described in the model should not be made independently of each other. For instance, a choice of a large processing repertoire such as MES-3 may not be sensible if the interchange repertoire is BL within the 8-bit code structure. It is recommended that, if possible, the same repertoire is used throughout.

If this is not possible, the procurer must specify appropriate transformation, based on the user requirements. This may be done by analysing the interfaces between the character handling functions in the model.

• Processing to Output. This may well become a common problem when the multi-byte UCS is used as the processing repertoire and the data stream received and processed makes extensive use of that repertoire. In the short term, there may be font restrictions which do not allow certain characters to be rendered on output. Here a transformation mechanism (see below) is needed which transforms unsupported characters into an output form that can be recognised

• Processing to Interchange. Here, either an a priori agreement between the communicating systems will be in place concerning the interchange repertoire and code structure, or some protocol negotiation will take place. The processing to interchange transformation requirement needs to be examined at two levels. First, there may be a mismatch of the repertoires. In this case, some repertoire transformation function will be needed, and the user requirement must then state whether this should be reversible or not. Secondly, a code transformation function may be needed since the processing code structure is likely to be based on some manufacturer specification whilst the interchange code structure is likely to be standardised. See clause 10.4.

• Interchange to Processing. This is a complement of the preceding case.

Annexes

Annex A 8-bit Character Sets

Annex B The Universal Character Set (UCS)