UTF-8 in HDF5

Return to “Advanced Topics.”

# Using UTF-8 Encoding in HDF5 Applications

## Introduction

Text and character data are often discussed as though text means ASCII text. We even go so far as to call a file containing only ASCII text a plain text file. This works reasonably well for English (though better for American English than British English), but what if that plain text file is in French, German, Chinese, or any of several hundred other languages? This document introduces the use of UTF-8 Unicode, a much more extensive and flexible character set that can faithfully represent any of those languages.

This document assumes a working familiarity with UTF-8 Unicode (UTF-8). Any reader who is unfamiliar with UTF-8 encoding should read the Wikipedia UTF-8 article (`https://en.wikipedia.org/wiki/UTF-8`) before proceeding; it provides an excellent primer.

For our context, the most important UTF-8 concepts are:

- Multi-byte and variable-size character encodings
- Limitations of the ASCII character set
- Risks associated with the use of the term plain text
- Representation of multiple language alphabets or characters in a single document

More specific technical details will only become important if they affect the specifics of your application design or implementation.

## How and Where May UTF-8 Be Used in HDF5?

HDF5 uses characters in object names (which are actually link names, but that’s a story for a different article), dataset data, attribute names, and attribute data. Though the mechanisms differ, you can use either ASCII or UTF-8 character sets in all of these situations.

### Object and Attribute Names

By default, HDF5 creates object and attribute names with ASCII character encoding. An object or attribute creation property list setting is required to create object names with UTF-8 characters. This uses the function `H5Pset_char_encoding`, which sets the character encoding used for object and attribute names.

For example, the following call sequence could be used to create a dataset with its name encoded with the UTF-8 character set:

        lcpl_id = H5Pcreate(H5P_LINK_CREATE) ;
        error =   H5Pset_char_encoding(lcpl_id, H5T_CSET_UTF8) ;
        dset_id = H5Dcreate2(group_id, "datos_ñ", dtype_id, dspace_id,
                  lcpl_id, H5P_DEFAULT, H5P_DEFAULT) ;

If the character encoding of an object name is unknown, the combination of an `H5Dget_create_plist` call and an `H5Pget_char_encoding` call will reveal that information.

### Character Datatypes in Datasets and Attributes

Like object names, HDF5 character data in datasets and attributes is encoded as ASCII by default. Setting up attribute or dataset character data to be UTF-8-encoded is accomplished while defining the attribute or dataset datatype. This makes use of the function `H5Tset_cset`, which sets the character encoding to be used in building a character datatype.

For example, the following commands could be used to create an 8-character, UTF-8 encoded, string datatype for use in either an attribute or dataset:

        utf8_8char_dtype_id = H5Tcopy(H5T_C_S1) ;
        error =    H5Tset_cset(utf8_8char_dtype_id, H5T_CSET_UTF8) ;
        error =    H5Tset_size(utf8_8char_dtype_id, "8") ;

If a character or string datatype’s character encoding is unkonwn, an `H5Tget_cset` call can be used to determine that.

## Caveats, Pitfalls, and Things to Watch For

[ This section remains under development; it will be posted as soon as it is available. — FMB ]

Programmers who are accustomed to using ASCII text without accommodating other text encodings will have to be aware of certain common issues as they begin using UTF-8.

### Cross-platform Portability

Since the HDF5 Library handles datatypes directly, UTF-8 encoded text in dataset and attribute datatypes in an HDF5 file should work transparently across platforms. The same should be true of handling names of groups, datasets, committed datatypes, and attributes within a file.
Is this actually true? On Windows systems?
The platform-specific content, here and below, is sketchy and tentative and must be reviewed by someone who understands the issues and platform differences.

Be aware, however, of system or application limitations once data or other information has been extracted from an HDF5 file. The application or system must be designed to accommodate UTF-8 if the information is then used elsewhere in the application or system environment.

Data from a UTF-8 encoded HDF5 datatype, in either a dataset or an attribute, that has been established within an HDF5 application should “just work” within the HDF5 portions of the application.

### Filenames

Since file access is a system issue, filenames do not fall within the scope of HDF5’s UTF-8 capabilities; filenames are encoded at the system level.

### The Plain Text Illusion

Beware the use of the term plain text. Plain text is at best ambiguous, but often misleading. Many will assume that plain text means ASCII, but plain text German or French, for example, cannot be represented in ASCII. Plain text is only unambiguous in the context of English (and even then can be problematic!).

### Storage Size

Programmers and data users accustomed to working strictly with ASCII data generally make the reasonable assumption that 1 character, be it in an object name or in data, requires 1 byte of storage. This equation does not work when using UTF-8 or any other Unicode encoding. With Unicode encoding, number of characters is not synonymous with number of bytes. One must get used to thinking in terms of number of characters when talking about content, reserving number of bytes for discussions of storage size.

When working with Unicode text, one can no longer assume a 1:1 correspondence between the number of characters and the data storage requirement.

### System Dependencies

Linux, Unix, and similar systems generally handle UTF-8 in correct and predictable ways. There is an apparent consensus in the Linux community that “UTF-8 is just the right way to go.”

Mac OS also generally handles UTF-8 correctly.
But must investigate.

Windows systems internally use a different Unicode encoding (UCS-2, discussed in this UTF-16 article).
What’s the appropriate thing to say about UTF-8 on Windows?
Do we know that “A carefully designed HDF5 application using UTF-8 encoding within an HDF5 file can be expected to function as expected.”
I have seen references implying that “Windows has the reputation of a somewhat schizophrenic approach to text handling.”
Have we seen a situation where Windows silently used UCS-2 or UTF-16 when UTF-8 had been specified in an HDF5 application? Have we seen situations where HDF5 app[lications successfully use UTF-s encoding on Windows?

## Common Characters in UTF-8 and ASCII

One interesting feature of UTF-8 and ASCII is that the ASCII character set is a discrete subset of the UTF-8 character set and where they overlap, the encodings are identical. This means that a character string consisting entirely of members of the ASCII character set can be encoded in either ASCII or UTF-8, the two encodings will be indistinguishable, and the encodings will require exactly the same storage space.

## See Also

For object and attibute names: `H5Pset_char_encoding` `H5Pget_char_encoding` | For dataset and attribute datatypes: `H5Tset_cset` `H5Tget_cset`
UTF-8 article on Wikipedia

Return to “Advanced Topics.”