Symbian Os Localization

SDN++

LOCALIZATION

3

Localization part of the SDN++ series 1st Edition, 07/08 Published by: Symbian Software Limited 2-6 Boundary Row Southwark London SE1 8HP UK www.symbian.com Trademarks, copyright, disclaimer ‘Symbian,’ ‘Symbian OS,’ and other associated Symbian marks are all trademarks of Symbian Software Ltd. Symbian acknowledges the trademark rights of all third parties referred to in this material. © Copyright Symbian Software Ltd 2008. All rights reserved. No part of this material may be reproduced without the express written permission of Symbian Software Ltd. Symbian Software Ltd makes no warranty or guarantee about the suitability or the accuracy of the information contained in this document. The information contained in this document is for general information purposes only and should not be used or relied upon for any other purpose whatsoever. Compiled by: Will Bamberg Managing Editor: Ashlee Godwin Reviewed by: Tim Band Chris Cooper Kai Duan Antony Edwards Freddie Gjertsen James Heginbottom Jasdeep Sawhney Renota Schoepp Jo Stichbury

An early draft of this booklet was reviewed by Chris Cooper, who passed away in early July 2008. He will be sadly missed for his contribution to the Text and Internationalization team in Symbian, and to the character of the company itself.

4

Contents INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 TEXT SUPPORT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 SCRIPT SUPPORT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 CONVERTING THE INPUTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 FONT SUPPORT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 LOCALES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 WHAT IS A LOCALE? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 LOCALE MODEL OVERVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 SUPPLIER INTERFACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 CLIENT INTERFACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 SELECTED LOCALE COMPONENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 LOCALE LIFECYCLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 CHECKLIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 GLOSSARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5

Introduction Symbian smartphones are used in many different countries, currently selling through more than 250 major network operators worldwide. There is no single type of ‘smartphone user’ and it is important for phone manufacturers to be able to tailor their devices for a number of different languages, regions, and cultures. Besides the fundamental differences of language, scripts, and reading direction, there are a number of additional characteristics that must be varied, such as the way names, dates, numbers, and currencies are formatted. Symbian OS, and the application layers above it, can be readily adapted for those variations. For example, the Contacts application does not need to be changed when it ships in two phones, each built to use a different sort order for the contact names. Localization is analogous to base porting. Symbian OS does not make assumptions about the specific hardware it runs on, but runs unchanged on top of an adaptation layer that needs to be reimplemented as part of the base port to a different hardware platform. Similarly, Symbian OS does not make assumptions about geography, culture, or language. It supplies a number of adaptation mechanisms, and device creators supply the data and algorithms that vary from one market to another. What’s the difference between Internationalization and Localization? Internationalization is the up-front design and implementation of software to allow it to run unchanged in various locales. Localization is the adaptation of a software system for a specific region or language. It typically adds new components specific to the region in question. It may help to think of them in terms of the development roles. The software developers inside Symbian, who design and implement the features of Symbian OS for supporting different locales, are internationalization engineers. Developers working inside device creation companies, who create phones for different regions, are localization engineers. The terms internationalization and localization are frequently abbreviated to i18n (where 18 stands for the number of letters between the i and the n), and L10n respectively. The capital L on L10n helps to distinguish it from the lowercase i in i18n.

This booklet provides an overview of the tasks a device creator needs to consider when localizing Symbian OS. It divides into two sections. The first section describes text processing issues that may arise when creating a new language variant. The second describes the support that the operating system provides for sets of user preferences, such as date format, that tend to vary from one language or geographic region to another, which are usually referred to as locales. Like all specialized technical domains, this one includes a significant amount of domainspecific terminology. Brief definitions of such terms are given where they are used, but the definitive reference is the Unicode glossary, at www.unicode.org/glossary.

6

Text Support This section considers issues of text support when creating a new language variant. The following must be guaranteed: • Symbian OS has support for rendering and editing the script • all text input to Symbian OS is converted into the Unicode UTF-16 encoding used internally, meaning that: - the appropriate character convertors are included to convert text entering the device from the outside world - the appropriate Front-end Processors (FEPs) are included for scripts that require specialized input methods (FEPs are explained in more detail in the section ‘Frontend Processors,’ later in this booklet) • suitable fonts are included in Symbian OS.

Figure 1 shows an extremely simplified view of the text processing components in Symbian OS. As the section ‘Converting The Inputs’ describes, text enters the system either from direct user input, or from the world outside the device (for example, in an email). Text entering the text processing subsystem must be in UTF-16. UTF-16 is a standard for encoding Unicode, defined in RFC 2781. The Symbian OS text formatting code takes as input: • the UTF-16 encoded text • formatting instructions (for example, the font to use, the font size, and which effects to apply). The Symbian OS text formatting code fetches the abstract character properties for the text (such as whether we are allowed to break a line at a given character) from the base Unicode support, and the physical dimensions of the text from the text rendering component. Then it uses all this information to calculate how to lay out the text. The text rendering component maps groups of characters onto glyphs for the purposes of measuring and drawing the text (a glyph is the representation in a font of a piece of text such as a character, including its dimensions and associated bitmap). The mapping between characters and glyphs is not always straightforward, so the text rendering code needs to work out how a group of characters resolves to a group of glyphs, and then retrieve them from the font.

7

Figure 1: Text processing subsystem. Device creators must consider three aspects when localizing Symbian OS for a different writing system (script): • basic script support: ensuring that the version of Symbian OS you are taking supports the script in question • converting the inputs: ensuring that text entering the text processing components has been converted into UTF-16 encoded Unicode • font support: supplying the correct fonts.

Script Support Some scripts are not supported by all releases of Symbian OS. For example, versions of Symbian OS prior to v9.2 did not support any Indic writing systems. The concept of ‘support’ for a script is really a kind of shorthand for supporting a number of specific use cases such as correct rendering and intuitive editing of the script.

8 This section gives an overview of some of the fundamental elements in supporting such use cases, but cannot pretend to be definitive. Even where one device creator has successfully shipped devices localized to use a specific script, this is no guarantee that the support offered by Symbian OS will be acceptable, as it is, to all device creators. Each may have subtly different requirements, for example, how cursor movement in bidirectional text should work. But it should give a good approximation of what is possible with the different releases.

Unicode Character Database Support Unicode defines a set of characters and assigns a code point and a corresponding collection of properties to each one. The properties include, for example, the character’s bidirectional class or line breaking properties. The complete set of code points and properties is called the Unicode Character Database (UCD) (www.unicode.org/ucd). Symbian OS stores the UCD internally in a table. The text formatting code retrieves the character properties from the table and uses this information to determine, for example, the directionality of the text or where line breaks are allowed. So for the platform to support a script its internal copy of the UCD needs to contain code points for all the script’s characters. Symbian OS releases from v9.1 to v9.3 contain the version of the UCD from Unicode 3.0.0, so any scripts added to the standard after 3.0.0 are not supported by those versions of the platform. From v9.4, Symbian OS contains a few extra code points needed for the following Indic scripts: Devanagari, Gujarati, Bengali, Gurmukhi, Tamil, and Kannada. Most notably, Symbian OS does not yet support any code points outside the Basic Multilingual Plane: that is, code points above FFFF, which therefore need to be encoded using a ‘surrogate pair’ consisting of two 16-bit values. Scripts containing characters mapped to code points above FFFF (also known as supplementary characters) are not currently supported.

Script-Specific Algorithm Support Apart from simply understanding the code points, many scripts have specific behavior which requires the text handling code to implement specific algorithms. Table 1 summarizes the Symbian OS releases in which support for such scripts was added: Symbian OS release

Script

v6.0

Japanese

v6.0

Chinese

v7.0s

Arabic

v7.0s

Hebrew

v8.0

Thai

v9.2

Vietnamese

v9.2

Devanagari

v9.4

Gujarati

v9.4

Kannada

Table 1: Support for writing systems added by Symbian OS version.

9 Languages requiring bidirectional support Arabic and Hebrew are interesting because they are written right-to-left, so implementing support for them means supporting the Unicode Bidirectional Algorithm (www.unicode.org/reports/tr9).

Languages with complex shaping requirements Thai, Devanagari, Gujarati, and Kannada are interesting because the visual representation (shape and placement) of a given letter or diacritic depends on the syllable in which it occurs, so they require the rendering and formatting code to understand where the divisions are between syllables. For Thai, which Symbian implements according to the WTT2.0 specification (www.inet.co.th/cyberclub/trin/thairef/wtt2/char-class.pdf), the rules are simple enough for it to be practical for the font file to include different glyphs for the different variant forms of each character, and for the rendering code to select the appropriate glyph based on the syllable in which it appears. But for the other scripts the rules are sufficiently complex that this is no longer very practical, and so the solution in these cases is to use OpenType fonts (partners.adobe.com/public/developer/opentype/index_spec.html) in combination with the open source ICU Layout Engine (www.icu-project.org/userguide/layoutEngine.html) to shape the glyphs based on their context. The ICU Layout Engine was integrated into Symbian OS in v9.2.

Converting The Inputs Internally, all text processing in Symbian OS assumes UTF-16 encoded Unicode text. So, any input to the text-processing subsystem must be in UTF-16 encoded Unicode. Input to the textprocessing subsystem comes either from the user (for example, in the form of keystrokes), in which case it is converted using a Front-end Processor (FEP), or it comes from some other computer (for example, a mail server), in which case it is converted using a Character Convertor (Charconv) plug-in.

Charconv The Character Conversion component, Charconv, is a plug-in framework for converting text between some other character encoding (or charset), referred to as the ‘foreign’ encoding, and the native UTF-16 encoding. Each convertor implements conversion between a single foreign character encoding and UTF-16, and is identified by UID. The UIDs are listed in the header file charconv.h. Symbian OS supplies: • A number of convertors built into the framework. These are convertors that the large majority of language variants are expected to need. • A group of additional convertors separately as ECOM plug-ins. These are convertors that are only needed for certain variants. Some convertors, such as those mapping Unicode, are universal, but most are intended to encode specific scripts or groups of scripts. Part of localizing the device for a new script involves ensuring that the device contains convertors for all the encodings in which the new language is commonly encoded. This may mean simply ensuring that the correct Symbiansupplied plug-ins are included, but in some cases the device creator may need to write additional plug-ins.

Table 2 summarizes the convertors supplied by Symbian OS. A § symbol is used to indicate

10 the release in which each first appeared if it was not available in the first EKA2 release of Symbian OS (v9.1). Each of the scripts shown is a plug-in, unless indicated by a ‡ symbol.

Name

Target script

Name

ISO 8859-6

Arabic

J5 DoCoMo

ISO 8859-13

Baltic

Shift-JIS DoCoMo

ISO 8859-14

Celtic

ISO 8859-2

Central European

GB2312

Target script

J5 KDDI (§ Symbian OS v9.3) Shift-JIS KDDI (§ Symbian OS v9.3)

Japanese encoding for DoCoMo Japanese encoding for DoCoMo Japanese encoding for KDDI Japanese encoding for KDDI

Chinese (Simplified)

ISO 8859-10

Nordic

GBK

Chinese (Simplified)

ISO 8859-4

Northern European

HZ

Chinese (Simplified)

ISO 8859-3

South European

Big5

Chinese (Traditional)

ISO 8859-9

Turkish

GB12345

Chinese (Traditional)

UTF-7 ‡

Universal (Unicode)

ISO 8859-5

Cyrillic

UTF-8 ‡

Universal (Unicode)

ISO 8859-7

Greek

UCS-2

Universal (Unicode)

ISO 8859-8

Hebrew

eucJP

Japanese

ISO 2022-JP

Japanese

ISO 2022-JP1

Japanese

JIS

Japanese

JIS X 0201

Japanese

JIS X 0208

Japanese

JIS X 0212

Japanese

Universal (Unicode) for IMAP Universal (Unicode) Java UTF-8 ‡ for Java Western European / CP-1252 ‡ US Western European / ISO 8859-1 ‡ US Western European / ASCII ‡ US Western European / SMS 7-bit ‡ US Western European / ISO 8859-15 US CP 850 Western European / (§ Symbian OS v9.2) US IMAP UTF-7 ‡

Table 2: Character convertors supplied by Symbian.

11 For example, a device to ship in Russia would need the ISO 8859-5 Cyrillic plug-in, but the device creator would also need to write a Windows-1251 plug-in, as that encoding is more widely used. Similarly, for a device to ship in Thailand the device creator would need to write an ISO 8859-11 convertor, as this is also not supplied by Symbian. Also note that the Shift-JIS convertor needs special attention. Different operators use different versions of the standard, and so two variants are produced: one for NTT DoCoMo and another for KDDI. Both versions of the standard use the same IANA name, so the convertors use the same UID and are not allowed to coexist in the same device. Documentation explaining how to build the correct Shift-JIS plug-in can be found in the Symbian OS codebase in the following location: \syslibs\charconv\plugins\documentation\shift-jis_howto.doc. We can distinguish two classes of Charconv plug-in: • Those where the foreign character set is small enough that the conversion is implemented as a lookup table: in this case the source code can be generated from specially formatted text files. • Those where the foreign character set is too large to implement the conversion as a lookup: then the conversion is defined by an algorithm (for example, UTF-8) and the source file must be handwritten. The Symbian Developer Library documentation contains detailed documentation on how to implement a Charconv plug-in (www.symbian.com/developer/techlib/v9.3docs/doc_source/ToolsAndUtilities93). Documentation for the tool used to generate mapping tables can be found in the Symbian OS codebase in the following location: \syslibs\charconv\plugins\documentation\cnvtool.rtf.

Front-end Processors Front-end Processors (FEPs) are plug-ins that convert user input into UTF-16 encoded input. Typical FEPs implement handwriting recognition systems or predictive text systems. Their relevance to script support is that certain scripts contain too many characters to fit on a device’s keyboard, so other input techniques are needed. The most obvious example is Chinese, in which the most common ways to implement text entry are Pinyin and handwriting recognition. When dealing with Arabic or Hebrew input, care should be taken to ensure the correct use of Bidirectional Ordering Control characters in the resulting input to the text processing subsystem (www.unicode.org/reports/tr9). This is particularly important when mixing Right-toLeft and Left-to-Right text.

Font Support The rendering code works out which sets of glyphs a set of Unicode characters maps to, and fetches the glyphs from the font in use. In order for a device to support a specific script, certain glyphs must be supported. Symbian publishes a font specification that documents these and the glyph codes that must be used to identify them. It can be found in the Symbian OS codebase in the following location: \graphics\fonts\Documentation\Symbian_OS_Font_Specification.doc.

12

Locales This section describes the support Symbian OS provides for locales, and includes a description of: • what a locale is • how to provision a device with new locales • which locale elements need special attention • how the locale is initialized, and how changes to it propagate through the system. Although the description here is specific to Symbian OS v9.4, it is largely valid for all versions of Symbian OS from v9.1 onwards.

What is a Locale? A locale is defined in Symbian OS as a set of data values that vary according to language and geographic region, for example, the names of the days of the week, the currency symbol, and the rules for sorting strings (collation). The locale settings include the language ID itself, which is a TLanguage enumeration value used to determine which localized resource file is loaded for an application when the application (or the application framework acting on its behalf ) calls BaflUtils::NearestLanguageFile(). Symbian OS also defines four locale ‘aspects,’ or groups of settings, and allows client code to load individual aspects in isolation from each other. The defined aspects are: • language settings, such as the names of the days of the week • time and date settings, such as date format • region settings, such as the currency symbol • collation settings, which determine the sort order for strings containing text. The reason for these distinctions is that it should be possible for certain subgroups of locale settings to vary independently of each other. For example, the desired language should be able to vary independently of the currently selected region.

Locale Model Overview Symbian’s locale support implements the mapping between two interfaces, as can be seen in Figure 2: • the client interface, which allows code on the device to retrieve and change locale settings • the supplier interface: the binary interface that must be implemented by suppliers of locale DLLs.

13

Application Application Application

Settings Application Loads locale data

Retrieves locale settings Client interface

Localization FW

Loads locale data using Supplier interface

Licensee code Symbian code

Implement interface

Localization DLL 1

Localization DLL 2

Figure 2: Locale model overview.

14 Supplier Interface Locale DLLs Each locale is supplied by the device creator and is packaged as a polymorphic interface DLL. Each DLL must implement the binary interface defined in eloclu.def. This is an implementation of the static functions in the Locl class, declared in localise.h: class Locl { public: IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static IMPORT_C static };

TLanguage Language(); TBool UniCode(); void LocaleData(SLocaleData *aLocale); const TLocaleText* CurrencySymbol(); const TLocaleText* ShortDateFormatSpec(); const TLocaleText* LongDateFormatSpec(); const TLocaleText* TimeFormatSpec(); const TFatUtilityFunctions* FatUtilityFunctions(); const TLocaleText* const *DateSuffixTable(); const TLocaleText* const *DayTable(); const TLocaleText* const *DayAbbTable(); const TLocaleText* const *MonthTable(); const TLocaleText* const *MonthAbbTable(); const TLocaleText* const *AmPmTable(); const TLocaleText* const *MsgTable(); const LCharSet *CharSet(); const TUint8 *TypeTable(); const TLocaleText* UpperTable(); const TLocaleText* LowerTable(); const TLocaleText* FoldTable(); const TLocaleText* CollTable();

The DLL is conventionally named ‘elocl.XYZ’ where ‘XYZ’ is taken from the TLanguage enumeration, defined in e32lang.h. Thus, the locale for UK English is named elocl.01 and that for Arabic is named elocl.37. If TLanguage does not contain a value for the locale you want to create, post a request on the SDN++ discussion forum for Symbian APIs (developer.symbian.com/forum/forum.jspa?forumID=8), stating the locale that you need. Symbian’s LOCE32 component contains some reference locale DLLs. The simplest way to create a locale is to copy and adapt one of these. The documentation of LOCE32 dates from Symbian OS v6, but is still quite useful, and may be found in Symbian’s codebase under \base\loce32\ongoing\doc.

The Default Locale Whenever Symbian OS boots up, the initialiselocale executable runs. This reads a configuration file implemented as a central repository keyspace, and uses it to populate the system locale. Subsequently, initialiselocale monitors the current system locale and when it changes it writes the new values into the central repository so the values persist across future re-boots.

15 Device creators that wish to make use of initialiselocale need to supply a copy of the central repository keyspace that contains the initial default values for the locale. The UID for this keyspace is 0x1020E4D3, and a sample keyspace is provided along with the initialiselocale component, which can be found in the \syslibs\bafl\initlocale\data\ directory of the Symbian OS codebase. The keyspace contains: • four string values, one for each aspect, each of which contains the name of a locale DLL from which to load that aspect • a series of customizable individual settings.

Client Interface Retrieving locale data Client code can retrieve locale settings using the TExtendedLocale class. To use this class link against EUser.lib and include e32std.h. To initialize a locale object to the current system settings the client needs to execute code like this: TExtendedLocale locale; locale.LoadSystemSettings(); The client can then retrieve the values: TPtrC currencySymbol = locale.GetCurrencySymbol();

Setting locale data While many applications will want to retrieve locale settings, it is expected that only a settings application provided by the UI vendor will need to change the settings. Individual settings can be changed manually with code such as this: TExtendedLocale locale; locale.LoadSystemSettings(); locale.SetCurrencySymbol(currencySymbol); locale.SaveSystemSettings();

Setting all locale values from a locale DLL More probably, the settings application will want to set all the settings from a locale DLL. The following code can be used to load all the settings from a given locale DLL: TExtendedLocale locale; locale.LoadSystemSettings(); locale.LoadLocale(dllName); locale.SaveSystemSettings();

Setting all locale values in a single aspect from a locale DLL The following code can be used to load a single aspect: TExtendedLocale locale; locale.LoadSystemSettings(); locale.LoadLocaleAspect(ELocaleLanguageSettings), dllName); locale.SaveSystemSettings();

16 Note that not all locale settings belong to an aspect. FatUtilityFunctions is used only by the file server, and the collection of settings in SLocaleData is not part of any aspect, and can only be loaded by loading the entire locale.

Publishing the new values Note that without the call to SaveSystemSettings()in the examples above, the locale settings are not published to the system and remain local to this locale object. This distinction enables an application to use a collection of locale settings which differs from the system settings.

Detecting changes to the system locale To detect changes to the system locale you can use RChangeNotifier, declared in e32std.h, and look for its asynchronous Logon(TRequestStatus& aStatus) method to complete with a status containing the EChangesLocale bit flag (defined in e32const.h).

Selected Locale Components Collation Settings The supplier interface function Locl::CharSet() returns the collation settings to use for the current locale. Symbian OS supports the Unicode Collation Algorithm. The rules for collation defined by version 2.1.9 of the standard are encoded as constant static data in collate.h, and these are referenced by the default implementation of Locl::CharSet(). It is possible that a specific locale will require different sorting to that specified in the standard. This is most likely to be where the same characters are used in different languages and the different languages expect different sorting for them. If you need different sorting to that specified in the standard, then you need to generate a custom collation table using the COLTAB tool and build that into your locale. This process is documented in Symbian’s codebase under \base\loce32\ongoing\coltab.

FAT Utility Functions FAT Utility Functions definition The supplier interface function Locl::FatUtilityFunctions() returns a pointer to an object of type TFatUtilityFunctions (declared in localise.h). An instance of this structure corresponds to a specific local character encoding, and is a struct containing three function pointers: TConvertFromUnicodeL

Converts from UTF-16 to the local character set.

TConvertToUnicodeL

Converts from the local character set to UTF-16.

TIsLegalShortNameCharacter

Returns true if the character is legal in the local character set in question.

17 FAT Utility Functions purpose The purpose of this element is to enable Symbian OS to interact with FAT file systems in a variety of regions. For example, a Japanese user will want to use Japanese names for files and directories on their SD card, and this means the file system plug-in will need to convert file and directory names from the UTF-16 encoding used internally into an encoding conventional in Japan. This is exactly analogous to the use of Charconv explained previously: • When names originate in Symbian OS and act as input to the FAT file system, they need to be converted from Unicode to the local encoding. • When names are being read from the FAT device for use in Symbian OS they need to be converted to UTF-16 from the local encoding. So a command like MkDirL() will take a 16-bit name as an argument, and the FAT file system plug-in will convert this to an 8-bit legal short name before writing the directory entry:

Figure 3: MkDirL() interaction with locale settings. Correspondingly, commands to read directory entries convert the names read from the disk into UTF-16 encoded Unicode, using ConvertToUnicodeL().

Relationship to the locale We have seen that the character encoding to be used for FAT file and directory names has a close relationship with the locale in which the device is being used. But the relationship is not straightforward: in particular, we do not necessarily want the character encoding to change whenever the UI language or even the locale change, because this could invalidate, or at least change the names of, existing files on FAT file systems. The solution currently in place is: • On boot, the conversion functions are NULL (which is interpreted effectively as ASCII). • The first time a locale with a non-NULL pointer is loaded, that is set as the FAT Utility functions pointer, and its mapping is used subsequently.

18 • This pointer does not change until the next reboot, whether or not new locales containing different pointers are loaded.

Implementing Fat Utility Functions Symbian OS supplies implementations of mapping functions for a variety of character encodings in the component FATCharsetConv. However, these implementations are supplied as standalone plug-in DLLs, so the device creator needs to extract the function implementations and package them as part of the relevant locale DLLs. The process of packaging the functions inside locale DLLs is documented in detail in the document ‘SGL.TS0019.219 – FatCharSetConv Integration HowTo-2.doc,’ which may be found in Symbian’s codebase under \base\documentation\. It is anticipated that in future releases of the OS the file server will load FATCharsetConv plug-ins directly, and the device creator will no longer have to go through this step.

Locale Lifecycle The locale lifecycle involves the interaction of several different system components. Note that the lifecycle described here is the default one provided by Symbian: several of the system components, including initialiselocale and those involved in controlling the boot sequence, may be replaced by a device creator.

Boot Locale setup during boot is controlled by the EStart component and passes through three distinct phases.

First phase In the first phase EStart sets the locale values to some hard coded defaults:

Figure 4: Boot, first phase: hard coded defaults. Note that in this phase the locale properties are set directly, bypassing TExtendedLocale. After this phase EStart can mount local drives.

19 Second phase In the second phase, EStart initializes the HAL data, and then uses the ELanguageIndex HAL setting to load the initial locale:

Figure 5: Boot, second phase: locale loaded from HAL settings. HalSettings uses the file HAL.DAT to initialize the HAL settings. Once they are initialized EStart retrieves the language setting and uses that to construct the name of a locale DLL to load (for instance, ‘ELOCL.37’ if the language index returned by HAL is 37, for Arabic). It then loads the locale and sets it as the current system locale. Now EStart can mount removable drives.

Third phase In the third phase EStart runs SysStart, which runs initialiselocale to read the persisted locale settings from the central repository keyspace and publish them to the system. This is shown in Figure 6.

20

Figure 6 Boot, third phase: locale initialized from initialiselocale.

21 Finally, initialiselocale requests notification of any changes to the locale settings:

Figure 7: Boot, final phase: initialiselocale requests update notifications.

Persisting Changes When a client changes any of the currently selected locale values either individually, or by loading a locale aspect, or by loading an entire locale, the change notifier is triggered:

Figure 8: Changing system settings.

22 This causes initialiselocale to persist the new locale values to the central repository:

Figure 9: Persisting locale settings. So at the next boot, these values will be installed as the system settings.

Resetting the Locale Resetting the locale to the factory settings simply uses the central repository’s reset implementation, so the persisted copy of keyspace 0x1020E4D3 will be replaced with the original copy. The device creator should take care to set the keyspace metadata controlling the reset behavior, otherwise restoring the locale settings to the factory default will not function correctly.

Backup and Restore If the persisted settings change through backup and restore, they will take effect after the next reboot. Note that initialiselocale does not listen to restore notifications.

23

Checklist This section summarizes the main questions you need to ask when preparing a new localization of a Symbian OS-based device. Localization Checklist 1

Does Symbian have basic support for the writing system (script)?

2

Which character encodings are commonly used to encode the language?

2a

Does Symbian already supply convertors (Charconv plug-ins) for all these encodings, or will you need to create some?

3

Does the writing system impose any special input requirements, such as Pinyin or bidirectional control codes?

4

Do you have fonts that conform to Symbian’s font specification?

5

Which locales will you need to create?

5a

Will any of the locales need custom collation methods?

5b

Will any of the locales need conversion functions for FAT file systems?

5c

Will any of the locales need language identifiers not already defined in TLanguage?

5d

Which locale will be the default?



Glossary The most comprehensive resource available for this topic is the Unicode glossary, which can be found at www.unicode.org/glossary.

24

References Unicode Character Database www.unicode.org/ucd Unicode Bidirectional Algorithm www.unicode.org/reports/tr9 WTT 2.0 Thai Input and Output Methods www.inet.co.th/cyberclub/trin/thairef/wtt2/char-class.pdf OpenType partners.adobe.com/public/developer/opentype/index_spec.html ICU Layout Engine www.icu-project.org/userguide/layoutEngine.html How to create a Charconv plug-in (in the Symbian OS v9.3 Symbian Developer Library) www.symbian.com/developer/techlib/v9.3docs/doc_source/ToolsAndUtilities93

25

New from

Quick Recipes on Symbian OS This book aims to make it easier to develop applications by describing a number of common programming tasks and providing clear explanations of how to complete them. The recipes are divided by technology, including graphics, multimedia, location-based services, networking, telephony, connectivity, and messaging.

Games on Symbian OS: A Handbook for Mobile Development This book forms part of the Technology Series from Symbian Press. It describes the key aspects of the mobile games marketplace, with particular emphasis on creating games for smartphones based on Symbian OS v9.x.

Symbian Press: developer.symbian.com/press

26

from

Developing Software for Symbian OS, Second Edition This second edition of Developing Software for Symbian OS helps software developers new to Symbian OS to create smartphone applications. The original book has been updated for Symbian OS v9 and now includes a new chapter on application signing and platform security, and updates throughout for Symbian OS v9 and changes to the development environment.

Symbian OS C++ for Mobile Phones, Volume 3 This book will help you to become an effective Symbian OS developer, and will give you a deep understanding of the fundamental principles upon which Symbian OS is based.

27

Also from

The Symbian OS Architecture Sourcebook This book conducts a rapid tour of the architecture of Symbian OS and provides an introduction to the key ideas of object orientation (OO) in software, with a detailed exploration of the architecture of Symbian OS.

Mobile Python Mobile Python is a practical hands-on book that introduces the popular open source programming language Python to the mobile space. It teaches how to program your own powerful - and fun applications easily on Nokia smartphones based on Symbian OS and the S60 platform.

Symbian Press: developer.symbian.com/press

28

Also from

For all Symbian C++ developers: Symbian OS Communications Programming, 2nd Edition by Iain Campbell S60 Programming - A Tutorial Guide by Coulton & Edwards Symbian OS Explained by Jo Stichbury Symbian OS Internals by Jane Sales Symbian OS Platform Security by Craig Heath Smartphone Operating System Concepts with Symbian OS by Mike Jipping Accredited Symbian Developer Primer by Jo Stichbury & Mark Jacobs

29

Also from

Published Booklets Coding Standards Coding Tips Performance Tips Essential UIQ - Getting Started Getting to Market Getting Started Quick Recipes Taster Java ME on Symbian OS P.I.P.S Carbide.c++ v1.3 Data Sharing Tips Essential S60 - Developers’ Guide

Translated Booklets Chinese Japanese Korean

Spanish Russian Persian

30

Notes

31

Notes

SDN++

LOCALIZATION SDN++

Why?

What?

Where?

How?

Localization is the adaptation of a software system for a specific region or language. This booklet provides an overview of the tasks a device creator needs to consider when localizing the operating system. The first section describes text processing issues that may arise when creating a new language variant. The second describes the support that Symbian OS provides for sets of user preferences, such as date format, that tend to vary from one language or geographic region to another, which are usually referred to as locales. Localization is part of the SDN++ series, designed to provide information in a handy format to SDN++ developers. For further information about the booklets, or to make suggestions for future titles, please contact [email protected].

Symbian Press Symbian Press publishes books designed to communicate authoritative, timely, relevant and practical information about Symbian OS and related technologies. Information about the Symbian Press series can be found at developer.symbian.com/books