This is Info file libc.info, produced by Makeinfo version 1.68 from the
input file libc.texinfo.

INFO-DIR-SECTION GNU libraries
START-INFO-DIR-ENTRY
* Libc: (libc).                 C library.
END-INFO-DIR-ENTRY

   This file documents the GNU C library.

   This is Edition 0.09 DRAFT, last updated 28 Aug 1999, of `The GNU C
Library Reference Manual', for Version 2.2 Beta.

   Copyright (C) 1993, '94, '95, '96, '97, '98, '99 Free Software
Foundation, Inc.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU Library General Public License" is
included exactly as in the original, and provided that the entire
resulting derived work is distributed under the terms of a permission
notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the text of the translation of the section
entitled "GNU Library General Public License" must be approved for
accuracy by the Foundation.


File: libc.info,  Node: Generic Conversion Interface,  Next: iconv Examples,  Up: Generic Charset Conversion

Generic Character Set Conversion Interface
------------------------------------------

   This set of functions follows the traditional cycle of using a
resource: open-use-close.  The interface consists of three functions,
each of which implement one step.

   Before the interfaces are described it is necessary to introduce a
datatype.  Just like other open-use-close interface the functions
introduced here work using a handles and the `iconv.h' header defines a
special type for the handles used.

 - Data Type: iconv_t
     This data type is an abstract type defined in `iconv.h'.  The user
     must not assume anything about the definition of this type, it
     must be completely opaque.

     Objects of this type can get assigned handles for the conversions
     using the `iconv' functions.  The objects themselves need not be
     freed but the conversions for which the handles stand for have to.

The first step is the function to create a handle.

 - Function: iconv_t iconv_open (const char *TOCODE, const char
          *FROMCODE)
     The `iconv_open' function has to be used before starting a
     conversion.  The two parameters this function takes determine the
     source and destination character set for the conversion and if the
     implementation has the possibility to perform such a conversion the
     function returns a handle.

     If the wanted conversion is not available the function returns
     `(iconv_t) -1'.  In this case the global variable `errno' can have
     the following values:

    `EMFILE'
          The process already has `OPEN_MAX' file descriptors open.

    `ENFILE'
          The system limit of open file is reached.

    `ENOMEM'
          Not enough memory to carry out the operation.

    `EINVAL'
          The conversion from FROMCODE to TOCODE is not supported.

     It is not possible to use the same descriptor in different threads
     to perform independent conversions.  Within the data structures
     associated with the descriptor there is information about the
     conversion state.  This must not be messed up by using it in
     different conversions.

     An `iconv' descriptor is like a file descriptor as for every use a
     new descriptor must be created.  The descriptor does not stand for
     all of the conversions from FROMSET to TOSET.

     The GNU C library implementation of `iconv_open' has one
     significant extension to other implementations.  To ease the
     extension of the set of available conversions the implementation
     allows storing the necessary files with data and code in
     arbitrarily many directories.  How this extension has to be
     written will be explained below (*note glibc iconv
     Implementation::.).  Here it is only important to say that all
     directories mentioned in the `GCONV_PATH' environment variable are
     considered if they contain a file `gconv-modules'.  These
     directories need not necessarily be created by the system
     administrator.  In fact, this extension is introduced to help users
     writing and using their own, new conversions.  Of course this does
     not work for security reasons in SUID binaries; in this case only
     the system directory is considered and this normally is
     `PREFIX/lib/gconv'.  The `GCONV_PATH' environment variable is
     examined exactly once at the first call of the `iconv_open'
     function.  Later modifications of the variable have no effect.

     This function got introduced early in the X/Open Portability Guide,
     version 2.  It is supported by all commercial Unices as it is
     required for the Unix branding.  However, the quality and
     completeness of the implementation varies widely.  The function is
     declared in `iconv.h'.

   The `iconv' implementation can associate large data structure with
the handle returned by `iconv_open'.  Therefore it is crucial to free
all the resources once all conversions are carried out and the
conversion is not needed anymore.

 - Function: int iconv_close (iconv_t CD)
     The `iconv_close' function frees all resources associated with the
     handle CD which must have been returned by a successful call to
     the `iconv_open' function.

     If the function call was successful the return value is 0.
     Otherwise it is -1 and `errno' is set appropriately.  Defined
     error are:

    `EBADF'
          The conversion descriptor is invalid.

     This function was introduced together with the rest of the `iconv'
     functions in XPG2 and it is declared in `iconv.h'.

   The standard defines only one actual conversion function.  This has
therefore the most general interface: it allows conversion from one
buffer to another.  Conversion from a file to a buffer, vice versa, or
even file to file can be implemented on top of it.

 - Function: size_t iconv (iconv_t CD, char **INBUF, size_t
          *INBYTESLEFT, char **OUTBUF, size_t *OUTBYTESLEFT)
     The `iconv' function converts the text in the input buffer
     according to the rules associated with the descriptor CD and
     stores the result in the output buffer.  It is possible to call the
     function for the same text several times in a row since for
     stateful character sets the necessary state information is kept in
     the data structures associated with the descriptor.

     The input buffer is specified by `*INBUF' and it contains
     `*INBYTESLEFT' bytes.  The extra indirection is necessary for
     communicating the used input back to the caller (see below).  It is
     important to note that the buffer pointer is of type `char' and the
     length is measured in bytes even if the input text is encoded in
     wide characters.

     The output buffer is specified in a similar way.  `*OUTBUF' points
     to the beginning of the buffer with at least `*OUTBYTESLEFT' bytes
     room for the result.  The buffer pointer again is of type `char'
     and the length is measured in bytes.  If OUTBUF or `*OUTBUF' is a
     null pointer the conversion is performed but no output is
     available.

     If INBUF is a null pointer the `iconv' function performs the
     necessary action to put the state of the conversion into the
     initial state.  This is obviously a no-op for non-stateful
     encodings, but if the encoding has a state such a function call
     might put some byte sequences in the output buffer which perform
     the necessary state changes.  The next call with INBUF not being a
     null pointer then simply goes on from the initial state.  It is
     important that the programmer never makes any assumption on
     whether the conversion has to deal with states or not.  Even if
     the input and output character sets are not stateful the
     implementation might still have to keep states.  This is due to the
     implementation chosen for the GNU C library as it is described
     below.  Therefore an `iconv' call to reset the state should always
     be performed if some protocol requires this for the output text.

     The conversion stops for three reasons.  The first is that all
     characters from the input buffer are converted.  This actually can
     mean two things: really all bytes from the input buffer are
     consumed or there are some bytes at the end of the buffer which
     possibly can form a complete character but the input is
     incomplete.  The second reason for a stop is when the output
     buffer is full.  And the third reason is that the input contains
     invalid characters.

     In all these cases the buffer pointers after the last successful
     conversion, for input and output buffer, are stored in INBUF and
     OUTBUF and the available room in each buffer is stored in
     INBYTESLEFT and OUTBYTESLEFT.

     Since the character sets selected in the `iconv_open' call can be
     almost arbitrary there can be situations where the input buffer
     contains valid characters which have no identical representation
     in the output character set.  The behavior in this situation is
     undefined.  The *current* behavior of the GNU C library in this
     situation is to return with an error immediately.  This certainly
     is not the most desirable solution.  Therefore future versions
     will provide better ones but they are not yet finished.

     If all input from the input buffer is successfully converted and
     stored in the output buffer the function returns the number of
     non-reversible conversions performed.  In all other cases the
     return value is `(size_t) -1' and `errno' is set appropriately.
     In this case the value pointed to by INBYTESLEFT is nonzero.

    `EILSEQ'
          The conversion stopped because of an invalid byte sequence in
          the input.  After the call `*INBUF' points at the first byte
          of the invalid byte sequence.

    `E2BIG'
          The conversion stopped because it ran out of space in the
          output buffer.

    `EINVAL'
          The conversion stopped because of an incomplete byte sequence
          at the end of the input buffer.

    `EBADF'
          The CD argument is invalid.

     This function was introduced in the XPG2 standard and is declared
     in the `iconv.h' header.

   The definition of the `iconv' function is quite good overall.  It
provides quite flexible functionality.  The only problems lie in the
boundary cases which are incomplete byte sequences at the end of the
input buffer and invalid input.  A third problem, which is not really a
design problem, is the way conversions are selected.  The standard does
not say anything about the legitimate names, a minimal set of available
conversions.  We will see how this negatively impacts other
implementations, as is demonstrated below.


File: libc.info,  Node: iconv Examples,  Next: Other iconv Implementations,  Prev: Generic Conversion Interface,  Up: Generic Charset Conversion

A complete `iconv' example
--------------------------

   The example below features a solution for a common problem.  Given
that one knows the internal encoding used by the system for `wchar_t'
strings one often is in the position to read text from a file and store
it in wide character buffers.  One can do this using `mbsrtowcs' but
then we run into the problems discussed above.

     int
     file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail)
     {
       char inbuf[BUFSIZ];
       size_t insize = 0;
       char *wrptr = (char *) outbuf;
       int result = 0;
       iconv_t cd;
     
       cd = iconv_open ("UCS4", charset);
       if (cd == (iconv_t) -1)
         {
           /* Something went wrong.  */
           if (errno == EINVAL)
             error (0, 0, "conversion from `%s' to `UCS4' no available",
                    charset);
           else
             perror ("iconv_open");
     
           /* Terminate the output string.  */
           *outbuf = L'\0';
     
           return -1;
         }
     
       while (avail > 0)
         {
           size_t nread;
           size_t nconv;
           char *inptr = inbuf;
     
           /* Read more input.  */
           nread = read (fd, inbuf + insize, sizeof (inbuf) - insize);
           if (nread == 0)
             {
               /* When we come here the file is completely read.
                  This still could mean there are some unused
                  characters in the `inbuf'.  Put them back.  */
               if (lseek (fd, -insize, SEEK_CUR) == -1)
                 result = -1;
     
               /* Now write out the byte sequence to get into the
                  initial state if this is necessary.  */
               iconv (cd, NULL, NULL, &wrptr, &avail);
     
               break;
             }
           insize += nread;
     
           /* Do the conversion.  */
           nconv = iconv (cd, &inptr, &insize, &wrptr, &avail);
           if (nconv == (size_t) -1)
             {
               /* Not everything went right.  It might only be
                  an unfinished byte sequence at the end of the
                  buffer.  Or it is a real problem.  */
               if (errno == EINVAL)
                 /* This is harmless.  Simply move the unused
                    bytes to the beginning of the buffer so that
                    they can be used in the next round.  */
                 memmove (inbuf, inptr, insize);
               else
                 {
                   /* It is a real problem.  Maybe we ran out of
                      space in the output buffer or we have invalid
                      input.  In any case back the file pointer to
                      the position of the last processed byte.  */
                   lseek (fd, -insize, SEEK_CUR);
                   result = -1;
                   break;
                 }
             }
         }
     
       /* Terminate the output string.  */
       if (avail >= sizeof (wchar_t))
         *((wchar_t *) wrptr) = L'\0';
     
       if (iconv_close (cd) != 0)
         perror ("iconv_close");
     
       return (wchar_t *) wrptr - outbuf;
     }

   This example shows the most important aspects of using the `iconv'
functions.  It shows how successive calls to `iconv' can be used to
convert large amounts of text.  The user does not have to care about
stateful encodings as the functions take care of everything.

   An interesting point is the case where `iconv' return an error and
`errno' is set to `EINVAL'.  This is not really an error in the
transformation.  It can happen whenever the input character set
contains byte sequences of more than one byte for some character and
texts are not processed in one piece.  In this case there is a chance
that a multibyte sequence is cut.  The caller than can simply read the
remainder of the takes and feed the offending bytes together with new
character from the input to `iconv' and continue the work.  The
internal state kept in the descriptor is *not* unspecified after such
an event as it is the case with the conversion functions from the ISO C
standard.

   The example also shows the problem of using wide character strings
with `iconv'.  As explained in the description of the `iconv' function
above the function always takes a pointer to a `char' array and the
available space is measured in bytes.  In the example the output buffer
is a wide character buffer.  Therefore we use a local variable WRPTR of
type `char *' which is used in the `iconv' calls.

   This looks rather innocent but can lead to problems on platforms
which have tight restriction on alignment.  Therefore the caller of
`iconv' has to make sure that the pointers passed are suitable for
access of characters from the appropriate character set.  Since in the
above case the input parameter to the function is a `wchar_t' pointer
this is the case (unless the user violates alignment when computing the
parameter).  But in other situations, especially when writing generic
functions where one does not know what type of character set one uses
and therefore treats text as a sequence of bytes, it might become
tricky.


File: libc.info,  Node: Other iconv Implementations,  Next: glibc iconv Implementation,  Prev: iconv Examples,  Up: Generic Charset Conversion

Some Details about other `iconv' Implementations
------------------------------------------------

   This is not really the place to discuss the `iconv' implementation
of other systems but it is necessary to know a bit about them to write
portable programs.  The above mentioned problems with the specification
of the `iconv' functions can lead to portability issues.

   The first thing to notice is that due to the large number of
character sets in use it is certainly not practical to encode the
conversions directly in the C library.  Therefore the conversion
information must come from files outside the C library.  This is
usually done in one or both of the following ways:

   * The C library contains a set of generic conversion functions which
     can read the needed conversion tables and other information from
     data files.  These files get loaded when necessary.

     This solution is problematic as it requires a great deal of effort
     to apply to all character sets (potentially an infinite set).  The
     differences in the structure of the different character sets is so
     large that many different variants of the table processing
     functions must be developed.  On top of this the generic nature of
     these functions make them slower than specifically implemented
     functions.

   * The C library only contains a framework which can dynamically load
     object files and execute the therein contained conversion
     functions.

     This solution provides much more flexibility.  The C library itself
     contains only very little code and therefore reduces the general
     memory footprint.  Also, with a documented interface between the C
     library and the loadable modules it is possible for third parties
     to extend the set of available conversion modules.  A drawback of
     this solution is that dynamic loading must be available.

   Some implementations in commercial Unices implement a mixture of
these these possibilities, the majority only the second solution.  Using
loadable modules moves the code out of the library itself and keeps the
door open for extensions and improvements.  But this design is also
limiting on some platforms since not many platforms support dynamic
loading in statically linked programs.  On platforms without his
capability it is therefore not possible to use this interface in
statically linked programs.  The GNU C library has on ELF platforms no
problems with dynamic loading in in these situations and therefore this
point is moot.  The danger is that one gets acquainted with this and
forgets about the restrictions on other systems.

   A second thing to know about other `iconv' implementations is that
the number of available conversions is often very limited.  Some
implementations provide in the standard release (not special
international or developer releases) at most 100 to 200 conversion
possibilities.  This does not mean 200 different character sets are
supported.  E.g., conversions from one character set to a set of, say,
10 others counts as 10 conversion.  Together with the other direction
this makes already 20.  One can imagine the thin coverage these platform
provide.  Some Unix vendors even provide only a handful of conversions
which renders them useless for almost all uses.

   This directly leads to a third and probably the most problematic
point.  The way the `iconv' conversion functions are implemented on all
known Unix system and the availability of the conversion functions from
character set A to B and the conversion from B to C does *not* imply
that the conversion from A to C is available.

   This might not seem unreasonable and problematic at first but it is a
quite big problem as one will notice shortly after hitting it.  To show
the problem we assume to write a program which has to convert from A to
C.  A call like

     cd = iconv_open ("C", "A");

does fail according to the assumption above.  But what does the program
do now?  The conversion is really necessary and therefore simply giving
up is no possibility.

   This is a nuisance.  The `iconv' function should take care of this.
But how should the program proceed from here on?  If it would try to
convert to character set B first the two `iconv_open' calls

     cd1 = iconv_open ("B", "A");

and

     cd2 = iconv_open ("C", "B");

will succeed but how to find B?

   Unfortunately, the answer is: there is no general solution.  On some
systems guessing might help.  On those systems most character sets can
convert to and from UTF8 encoded ISO 10646 or Unicode text.  Beside
this only some very system-specific methods can help.  Since the
conversion functions come from loadable modules and these modules must
be stored somewhere in the filesystem, one *could* try to find them and
determine from the available file which conversions are available and
whether there is an indirect route from A to C.

   This shows one of the design errors of `iconv' mentioned above.  It
should at least be possible to determine the list of available
conversion programmatically so that if `iconv_open' says there is no
such conversion, one could make sure this also is true for indirect
routes.

