\input texinfo   @c -*-texinfo-*-
@c %**start of header
@setfilename utf8gen.info
@settitle utf8gen
@setchapternewpage odd
@c %**end of header

@macro utf
@w{UTF-8}
@end macro

@paragraphindent none

@copying
This manual describes @command{utf8gen}, a utility for converting Unicode
hexadecimal code points into @utf{} as printable characters for immediate
viewing and as byte sequences suitable for including in programs.

Copyright @copyright{} 2018 Paul Hardy

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, with no Front-Cover Texts and no Back-Cover Texts.
@end quotation

@end copying

@dircategory Text

@direntry
* utf8gen: (utf8gen).  A utility for converting hexadecimal numbers into @utf{}
@end direntry

@titlepage

@title utf8gen
@author Paul Hardy
@page
@vskip 0pt plus 1filll

@insertcopying

@end titlepage

@contents


@node Top, Introduction, (dir), (dir)

@menu
* Introduction:: General information
* Unicode:: Overview of Unicode and @utf{}
* Invoking @command{utf8gen}:: Common Use Cases for using @command{utf8gen}
* @command{utf8gen} Reference:: Detailed description of the @command{utf8gen} utility
@end menu


@node Introduction, Unicode, Top, Top
@chapter Introduction


This document describes some typical uses for @command{utf8gen}, a utility
to read ASCII hexadecimal numbers, interpret them as Unicode
code points, and output Unicode Transformation Format --
@w{8-bit} (@utf{}).

If you have questions, please email
@email{unifoundry@@unifoundry.com}.
You can check for the latest @command{utf8gen} news at
@code{http://unifoundry.com/utf8gen/}.


--- Paul Hardy (@email{unifoundry@@unifoundry.com})  2018


@node Unicode, Invoking @command{utf8gen}, Introduction, Top
@chapter Unicode

@menu
* Unicode Overview::
* Unicode Planes::
* UTF-8::
@end menu


@node Unicode Overview, Unicode Planes, , Unicode
@section Unicode Overview

Unicode arose out of a practical need for a common encoding
to represent all of the world's languages on computers.
It has grown rapidly over the past 20+ years to contain
more than 100,000 glyphs (characters).  These glyphs are
divided over multiple Unicode @dfn{planes}: @w{Plane 0} through
@w{Plane 16} (decimal), for a total of 17 planes.  Each plane
contains 64k @dfn{code points}, which in hexadecimal is
10000 code points.  @dfn{Code point} is a more general
term than @dfn{character}, because Unicode contains more
than just visible characters; for example, Unicode contains
various code points for indicating variation selection
for scripts that have multiple forms of a visible character.


@node Unicode Planes, UTF-8, Unicode Overview, Unicode
@section Unicode Planes


@w{Plane 0} contains most of the world's modern scripts.
Code points in this range are denoted as Unicode code points
U+0000 through U+FFFF, inclusive.  This plane is also known as
the Basic Multilingual Plane, or BMP.  The ASCII code points
are in the beginning of the BMP, from U+0000 through U+007F.
The BMP is almost entirely allocated --- there are hardly any
free code point ranges in the BMP for assigning new scripts.
Fortunately, Unicode has 16 more planes beyone @w{Plane 0.}

@w{Plane 1} contains many ancient scripts, and modern collections
that were not assigned to @w{Plane 0} (for example, emoji).  This
plane is also known as the Supplementary Multilingual Plane,
or SMP.  Unicode code points in the SMP are in the range
U+10000 through U+1FFFF, inclusive.

@w{Plane 2} is the Supplementary Ideographic Plane, or SIP.
It contains Chinese and Japanese ideographs that were not
included in @w{Plane 0.}  Unicode code points in @w{Plane 2}
are in the range U+20000 through U+2FFFF, inclusive.

These are the main planes with assigned visible characters.

@w{Plane 14} is the Supplementary Special-purpose Plane, or SSP.
Its code points are in the range U+E0000 through U+EFFFF.
This plane contains specialized tags and other designators.

Planes 15 and 16 are Private Use Area (PUA) planes.  They
can contain any user-defined characters and special-purpose
codes.  These planes span the Unicode range U+FFFFF through
U+10FFFF.


@node UTF-8, , Unicode Planes, Unicode
@section UTF-8


Thus the valid Unicode range is U+0000 through U+10FFFF,
inclusive.  Encoding the entire Unicode range takes from
one byte for the ASCII range to 21 bits to encode anything
in @w{Plane 16} (U+100000 through U+10FFFF).

A problem with transmitting these multi-byte numbers is
that different computer architectures order bytes in a
multi-byte word differently.  Today there are only two
common orderings: big-endian, where the largest byte is
stored first, and little-endian, where the smallest byte
is stored first.

When transmitting information between computers of
different architectures, a standard protocol had to
be defined.  The Unicode encoding that modern web
browsers use is called Unicode Transformation Format
-- @w{8-bit,} or @utf{}.  It has also become the standard
encoding for text documents that contain non-ASCII
characters.

@utf{} encoding has several desirable characteristics,
which are described briefly below. 

The first byte in a @utf{} encoded character begins with
a series of @samp{1}@tie{}bits, to indicate how many bytes
the character requires, except a one-byte character
starts with a @samp{0}@tie{}bit to designate the byte as
ASCII.  The  ASCII range, U+0000 through U+007F,
is encoded the same in @utf{}, as just one byte.
Each byte after the first in a multi-byte character
begins with the bits@tie{}@samp{10}.

The number of bytes in a @utf{} encoded Unicode
code point varies from one to four bytes.  Thus
it is an efficient encoding compared to one that
would transmit the same number of bytes for every
character across the entire Unicode range.

No single-byte @utf{} character will ever begin with
the pattern @samp{10}, as single-byte @utf{} characters
always begin with a @samp{0}@tie{}bit.  So string searching
functions can skip bytes within a @utf{} byte string
and if a byte currently being examined begins with
the bits@tie{}@samp{10}, the search function knows it is
past the beginning of a multi-byte character.

Unicode code points are published in @dfn{code charts},
available at @url{http://unicode.org/}.
These code charts number code points using
hexadecimal.  These hexadecimal numbers must be
converted to @utf{} for transmission on web pages,
storing in a text document, etc.  Hence the
creation of @command{utf8gen}.


@node Invoking @command{utf8gen}, @command{utf8gen} Reference, Unicode, Top
@chapter Invoking @command{utf8gen}

@menu
* Motivation::
* Printing a Character::
* Code Generation::
* Use Case Summary::

@end menu


@node Motivation, Printing a Character, , Invoking @command{utf8gen}
@section Motivation


This chapter provides examples of typical uses
for @command{utf8gen} for programmers and end-users.

I needed to generate hundreds of lines of source
code containing different @utf{} characters for
a set of programs.  My searches did not find
anything that performed the conversion as I wanted,
so I wrote @command{utf8gen}.

With the Unicode Standard specifying code point
assignments in hexadecimal, it was natural to
write software that took a hexadecimal number
as input.  There are numerous potential forms
of output, especially considering the formatting
syntax of different programming languages.  The
purpose of most of the options for @command{utf8gen}
is to select various output options.

@command{utf8gen} reads in hexadecimal numbers,
one per input line.  Each number can be followed
by a space and a miscellaneous string to the end
of the line.  That @dfn{remainder} string can
optionally be printed on output; more on that
later.


@node Printing a Character, Code Generation, Motivation, Invoking @command{utf8gen}
@section Printing a Character


The simplest thing an end-user might want to know
is whether their computer has a font that supports
a certain Unicode character.  The easiest way
to use @command{utf8gen} is interactively at a
terminal, typing in hexadecimal numbers and
looking at the character produced.  To do this,
run the command

@example
utf8gen -c -n
@end example

The @option{-c} option tells @command{utf8gen} to
print the input hexadecimal number as a Unicode
character on the screen.  The @option{-n} option
tells @command{utf8gen} to @emph{not} print the
@utf{} byte sequence as a set of formatted
numbers.  Just enter one hexadecimal number
in the range 0 through 10FFFF, inclusive,
one number per line.  When finished running
@command{utf8gen} interactively in this way,
end your input by typing @key{C-d}.


@node Code Generation, Use Case Summary, Printing a Character, Invoking @command{utf8gen}
@section Code Generation


@menu
* The Usefulness of Octal::
* Commenting Code::
* Remainder Strings::
* UTF-8 Output Format::
* Input and Output Files::
@end menu


@node The Usefulness of Octal, Commenting Code, , Code Generation
@subsection The Usefulness of Octal


If converting hexadecimal numbers into a form
that a programming language accepts, there are
many possiblilties.  For this reason, @command{utf8gen}
accepts format strings in the style of the C
@code{printf} function.  This was a natural
choice, as @command{utf8gen} is written @w{in C.}

With eight bits in a byte, and @utf{} encoded 
characters starting either with a @samp{0}@tie{}bit
for ASCII or with @samp{10} for all but the
first byte in a multi-byte sequence, it is
convenient to look at Unicode code point numbers
encoded as octal.  If a byte in a @utf{} byte
string begins with @samp{10}, this leaves six
bits for the remainder of the byte.  This is
conveniently viewed as two octal digits.

The default output of @command{utf8gen} is simply
the sequence of octal digits in a @utf{} character,
printed in the C style of a backslash followed
by three octal digits per byte.  This is handy
for a quick copy and paste of a single @utf{}
byte sequence into a program.

If using the C-style backslashed octal number
format, it can be reassuring to see what a
Unicode code point is in octal (at least it
was for me, when I first wrote the program
and was verifying its proper operation).
A simple way of doing this is to have
@command{utf8gen} echo the input hexadecimal
number you typed in as octal, and then
print the @utf{} representation.  To do
this, run a command of the form

@example
utf8gen -e "%03o = "
@end example

For example, if you enter the hexadecimal
number @kbd{2134} (the Unicode code point for
@samp{Script Small Letter O} in the
@samp{Letterlike Symbols} block,
@command{utf8gen} will generate this output:

@example
20464 = \342\204\264
@end example

The hexadecimal number 2134 is 20464 in octal.
Notice how two octal digits from the Unicode
code point appear in each @utf{} byte except
for the first byte.  The leading octal digit
of @samp{2} represents the leading two bits
@samp{10} in a @utf{} multi-byte sequence.
The first byte in a multi-byte @utf{} sequence
starts with a string of @samp{1}@tie{}bits indicating
how many bytes long the encoded character is.
In this case, the @utf{} representation of
U+2134 will take three bytes, so the first byte
begins with the bit string @samp{1110}.  That
corresponds to the first two octal digits
(@samp{34}) of the first byte in the sequence,@tie{}@samp{\342}.

Looking at the sequence @samp{\342\204\264}
again, it is easy to see the placement of
the octal representation of this Unicode
code point, 20464.  In this way, verifying
the proper conversion of the hexadecimal
Unicode code point to @utf{} is straightforward.


@node Commenting Code, Remainder Strings, The Usefulness of Octal, Code Generation
@subsection Commenting Code


Commenting code is of course useful, especially
when dealing with something as arcane as raw
@utf{} byte sequences.  @command{utf8gen} provides
various ways of doing this.  A couple of examples
should suffice to give you an idea of these
capabilities.

The simplest method for creating comments might
be to follow an octal sequence with the Unicode
code point in its canonical form.  The @option{-e}
option @emph{echoes} the input number to the output
using the format string that follows.  This will
accomplish that:

@example
utf8gen -e "/* U+%04X */ "
@end example

For the hexadecimal input number 2134, this
produces the output

@example
/* U+2134 */ \342\204\264
@end example

The expectation is that a programmer will be
able to use an editor that can take a string
like @samp{\342\204\264} and easily convert
it into a @dfn{print}-style command in the
programming language of choice.

It might be preferable to print the comment
after the @utf{} byte sequence.  The @option{-s}
option allows this by @dfn{swapping} the default
output string order.  For example, the command

@example
utf8gen -e " /* U+%04X */" -s
@end example

produces the output (again, using 2134 as
the input number) of

@example
\342\204\264 /* U+2134 */
@end example

It might even be useful to output the initial
hexadecimal number using two different bases.
This is accomplished with the @option{-E} option,
followed by the format string for echoing the
input number in two ways.  For example, the command

@example
utf8gen -E " /* U+%04X = 0%o */" -s
@end example

produces the output (with an input of @samp{2134})

@example
\342\204\264 /* U+2134 = 020464 */
@end example


@node Remainder Strings, UTF-8 Output Format, Commenting Code, Code Generation
@subsection Remainder Strings


One can only glean so much by looking at numbers
though.  A textual comment describing a Unicode
code point can also help.  @command{utf8gen} supports
printing free-form text following an initial
hexadecimal number followed by a space.  This is
done with the @option{-r} option, to print the
@dfn{remainder} of the input line, using
the format string that follows this option.

The Unicode Consortium makes various data files
available with a free use license.  The first
field is usually the Unicode code point in hexadecimal.
Remaining fields will contain information about
each code point.  For example, given the following
line of input:

@example
2134 SCRIPT SMALL O
@end example

This command

@example
utf8gen -e " /* U+%04X " -s -r "%s */"
@end example

will produce this output:

@example
\342\204\264 /* U+2134 SCRIPT SMALL O */
@end example

This can facilitate batch processing of large
portions of a Unicode data file.


@node UTF-8 Output Format, Input and Output Files, Remainder Strings, Code Generation
@subsection UTF-8 Output Format


@command{utf8gen} also allows specifying the
format of the encoded @utf{} bytes with the
@option{-u} option followed by a format string.
For example, suppose the programming language you
use will accept bytes in hexadecimal using the form
@code{\x} followed by a hexadecimal number.
If we take the previous example input line and
provide it to @command{utf8gen} with the comand

@example
utf8gen -u "\x%02x" -r " /* %s */" -s
@end example

this will produce the output

@example
\xe2\x84\xb4 /* SMALL SCRIPT O */
@end example

If the @option{-r} option is selected but there
is nothing after the hexadecimal number on an
input line, no remainder content will be printed.

Of course, you could also use the @option{-e}
or @option{-E} options to echo back the input number
in the desired output format(s) by adding it to
the command line.


@node Input and Output Files, , UTF-8 Output Format, Code Generation
@subsection Input and Output Files


This information can be extracted and provided as
an input file to @command{utf8gen} using the @option{-i}
option to specify an input file.  Output can be
written to a file using the @option{-o} option.


@node Use Case Summary, , Code Generation, Invoking @command{utf8gen}
@section Use Case Summary


The descriptions in this chapter give a brief
overview of all of the @option{utf8gen} options
and how they might be used in practice.

@command{utf8gen} tries to strike a balance between
the basics that a programmer might find useful
for bulk conversion of a large number of hexadecimal
Unicode code points versus creeping featurism.
While @command{utf8gen} won't write your program
for you, it can make the bulk conversion of
code points efficient.


@node @command{utf8gen} Reference, , Invoking @command{utf8gen}, Top
@chapter @command{utf8gen} Reference


@comment TROFF INPUT: .TH UTF8GEN 1 "2018 Jun 30"

@c @node utf8gen, , , @command{utf8gen} Reference
@c @section utf8gen
@c DEBUG: print_menu("@section")

@menu
* NAME::
* SYNOPSIS::
* DESCRIPTION::
* OPTIONS::
* EXAMPLES::
* FILES::
* AUTHOR::
* LICENSE::
* BUGS::

@end menu


@comment TROFF INPUT: .SH NAME

@node NAME, SYNOPSIS, , @command{utf8gen} Reference
@section NAME
@c DEBUG: print_menu("utf8gen NAME")

utf8gen @minus{} Generate UTF-8 output from hexadecimal input
@comment TROFF INPUT: .SH SYNOPSIS

@node SYNOPSIS, DESCRIPTION, NAME, @command{utf8gen} Reference
@section SYNOPSIS
@c DEBUG: print_menu("utf8gen SYNOPSIS")

@comment TROFF INPUT: .br
@comment .br
@b{utf8gen} [ [-e @i{format1}] | [-E @i{format2}] ] [-r @i{formatr}]
        [ [-u @i{utf8@t{_}format}] | -n] [-c] [-s]
        [-i @i{input@t{_}file}] [-o @i{output@t{_}file}]
@comment TROFF INPUT: .SH DESCRIPTION

@node DESCRIPTION, OPTIONS, SYNOPSIS, @command{utf8gen} Reference
@section DESCRIPTION
@c DEBUG: print_menu("utf8gen DESCRIPTION")

@comment TROFF INPUT: .B utf8gen
@b{utf8gen}
reads a list of hexadecimal ASCII values in the range
0 through 10FFFF, one per line, and prints the UTF-8 encoding
of that number as a Unicode code point.
@comment TROFF INPUT: .PP

Each input line must begin with a hexadecimal number.
A string may follow after that, which can be echoed to the
output as the "remainder" (see the @option{-r} option below).
The total input line length, including an ending newline,
is limited to 4096 bytes.
@comment TROFF INPUT: .SH OPTIONS

@node OPTIONS, EXAMPLES, DESCRIPTION, @command{utf8gen} Reference
@section OPTIONS
@c DEBUG: print_menu("utf8gen OPTIONS")

@comment TROFF INPUT: .TP 6

@c ---------------------------------------------------------------------
@table @code
@item @option{-c}
After the UTF-8 codes are printed, print a space followed by
the character that the hexadecimal code point represents.
@comment TROFF INPUT: .TP

@item @option{-e}
Echo the input code point in one format, using the
printf(3) format string @i{format1}.
@comment TROFF INPUT: .TP

@item @option{-E}
Echo the input code point in two formats, using the
printf(3) format string @i{format2}.
@comment TROFF INPUT: .TP

@item @option{-n}
Do @i{not} print the UTF-8 byte values.  This can be useful
if only the printed character itself is desired; see the @option{-c} option.
@comment TROFF INPUT: .TP

@item @option{-r}
Print the remainder of the input string after the initial
hexadecimal digits, using the printf(3) format string @i{formatr}.
@comment TROFF INPUT: .TP

@item @option{-s}
Swap the order of output: print the UTF-8 output portion first,
then print the input string portion.  This can be useful for
generating code containing a UTF-8 encoding followed by a
comment that contains the input hexadecimal digits.
@comment TROFF INPUT: .SH EXAMPLES

@item @option{-u}
Print the UTF-8 encoded value of the input hexadecimal number,
as numeric codes for each UTF-8 byte, using the printf(3)
format string @i{utf8@t{_}format}.  If no string is specified,
a default format of a backslash followed by three octal digits
is printed for each byte.
@comment TROFF INPUT: .TP

@end table

@c ---------------------------------------------------------------------

@node EXAMPLES, FILES, OPTIONS, @command{utf8gen} Reference
@section EXAMPLES
@c DEBUG: print_menu("utf8gen EXAMPLES")

@comment TROFF INPUT: .RS

@c ---------------------------------------------------------------------
@quotation
@comment TROFF INPUT: .PP

         @code{utf8gen -e "0x%04X " -u "\\%03o"}
@comment TROFF INPUT: .PP

         @code{utf8gen -E "U+%04x = 0%02o = "}
@comment TROFF INPUT: .PP

         @code{utf8gen -s -e "  /* U+%04X */" -u "\\%03o"}
@comment TROFF INPUT: .RE

@end quotation

@c ---------------------------------------------------------------------
@comment TROFF INPUT: .SH FILES

@node FILES, AUTHOR, EXAMPLES, @command{utf8gen} Reference
@section FILES
@c DEBUG: print_menu("utf8gen FILES")

Files contain lines that each begin with an ASCII hexadecimal
code in the valid Unicode range 0 through 10FFFF, inclusive.
This hexadecimal code may optionally be followed by a space
followed by an arbitrary string ending with a newline,
up to the limit of 4096 bytes per input line.
An example line could be the following (with no indent):
@comment TROFF INPUT: .PP

@comment TROFF INPUT: .RS

@c ---------------------------------------------------------------------
@quotation
41 Letter 'A'
@comment TROFF INPUT: .RE

@end quotation

@c ---------------------------------------------------------------------
@comment TROFF INPUT: .SH AUTHOR

@node AUTHOR, LICENSE, FILES, @command{utf8gen} Reference
@section AUTHOR
@c DEBUG: print_menu("utf8gen AUTHOR")

@comment TROFF INPUT: .B utf8gen
@b{utf8gen}
was written by Paul Hardy.
@comment TROFF INPUT: .SH LICENSE

@node LICENSE, BUGS, AUTHOR, @command{utf8gen} Reference
@section LICENSE
@c DEBUG: print_menu("utf8gen LICENSE")

@comment TROFF INPUT: .B utf8gen
@b{utf8gen}
is Copyright @copyright{} 2018 Paul Hardy.
@comment TROFF INPUT: .PP

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
@comment TROFF INPUT: .SH BUGS

@node BUGS, , LICENSE, @command{utf8gen} Reference
@section BUGS
@c DEBUG: print_menu("utf8gen BUGS")

No known bugs exist.

@bye
