This is Info file libc.info, produced by Makeinfo version 1.68 from the
input file libc.texinfo.

INFO-DIR-SECTION GNU libraries
START-INFO-DIR-ENTRY
* Libc: (libc).                 C library.
END-INFO-DIR-ENTRY

   This file documents the GNU C library.

   This is Edition 0.09 DRAFT, last updated 28 Aug 1999, of `The GNU C
Library Reference Manual', for Version 2.2 Beta.

   Copyright (C) 1993, '94, '95, '96, '97, '98, '99 Free Software
Foundation, Inc.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU Library General Public License" is
included exactly as in the original, and provided that the entire
resulting derived work is distributed under the terms of a permission
notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the text of the translation of the section
entitled "GNU Library General Public License" must be approved for
accuracy by the Foundation.


File: libc.info,  Node: Converting a Character,  Next: Converting Strings,  Prev: Keeping the state,  Up: Restartable multibyte conversion

Converting Single Characters
----------------------------

   The most fundamental of the conversion functions are those dealing
with single characters.  Please note that this does not always mean
single bytes.  But since there is very often a subset of the multibyte
character set which consists of single byte sequences there are
functions to help with converting bytes.  One very important and often
applicable scenario is where ASCII is a subpart of the multibyte
character set.  I.e., all ASCII characters stand for itself and all
other characters have at least a first byte which is beyond the range 0
to 127.

 - Function: wint_t btowc (int C)
     The `btowc' function ("byte to wide character") converts a valid
     single byte character C in the initial shift state into the wide
     character equivalent using the conversion rules from the currently
     selected locale of the `LC_CTYPE' category.

     If `(unsigned char) C' is no valid single byte multibyte character
     or if C is `EOF' the function returns `WEOF'.

     Please note the restriction of C being tested for validity only in
     the initial shift state.  There is no `mbstate_t' object used from
     which the state information is taken and the function also does
     not use any static state.

     This function was introduced in the second amendment of ISO C90 and
     is declared in `wchar.h'.

   Despite the limitation that the single byte value always is
interpreted in the initial state this function is actually useful most
of the time.  Most characters are either entirely single-byte character
sets or they are extension to ASCII.  But then it is possible to write
code like this (not that this specific example is very useful):

     wchar_t *
     itow (unsigned long int val)
     {
       static wchar_t buf[30];
       wchar_t *wcp = &buf[29];
       *wcp = L'\0';
       while (val != 0)
         {
           *--wcp = btowc ('0' + val % 10);
           val /= 10;
         }
       if (wcp == &buf[29])
         *--wcp = L'0';
       return wcp;
     }

   Why is it necessary to use such a complicated implementation and not
simply cast `'0' + val % 10' to a wide character?  The answer is that
there is no guarantee that one can perform this kind of arithmetic on
the character of the character set used for `wchar_t' representation.
In other situations the bytes are not constant at compile time and so
the compiler cannot do the work.  In situations like this it is
necessary `btowc'.

There also is a function for the conversion in the other direction.

 - Function: int wctob (wint_t C)
     The `wctob' function ("wide character to byte") takes as the
     parameter a valid wide character.  If the multibyte representation
     for this character in the initial state is exactly one byte long
     the return value of this function is this character.  Otherwise
     the return value is `EOF'.

     This function was introduced in the second amendment of ISO C90 and
     is declared in `wchar.h'.

   There are more general functions to convert single character from
multibyte representation to wide characters and vice versa.  These
functions pose no limit on the length of the multibyte representation
and they also do not require it to be in the initial state.

 - Function: size_t mbrtowc (wchar_t *restrict PWC, const char
          *restrict S, size_t N, mbstate_t *restrict PS)
     The `mbrtowc' function ("multibyte restartable to wide character")
     converts the next multibyte character in the string pointed to by
     S into a wide character and stores it in the wide character string
     pointed to by PWC.  The conversion is performed according to the
     locale currently selected for the `LC_CTYPE' category.  If the
     conversion for the character set used in the locale requires a
     state the multibyte string is interpreted in the state represented
     by the object pointed to by PS.  If PS is a null pointer, a static,
     internal state variable used only by the `mbrtowc' function is
     used.

     If the next multibyte character corresponds to the NUL wide
     character the return value of the function is 0 and the state
     object is afterwards in the initial state.  If the next N or fewer
     bytes form a correct multibyte character the return value is the
     number of bytes starting from S which form the multibyte
     character.  The conversion state is updated according to the bytes
     consumed in the conversion.  In both cases the wide character
     (either the `L'\0'' or the one found in the conversion) is stored
     in the string pointer to by PWC iff PWC is not null.

     If the first N bytes of the multibyte string possibly form a valid
     multibyte character but there are more than N bytes needed to
     complete it the return value of the function is `(size_t) -2' and
     no value is stored.  Please note that this can happen even if N
     has a value greater or equal to `MB_CUR_MAX' since the input might
     contain redundant shift sequences.

     If the first `n' bytes of the multibyte string cannot possibly form
     a valid multibyte character also no value is stored, the global
     variable `errno' is set to the value `EILSEQ' and the function
     returns `(size_t) -1'.  The conversion state is afterwards
     undefined.

     This function was introduced in the second amendment to ISO C90 and
     is declared in `wchar.h'.

   Using this function is straight forward.  A function which copies a
multibyte string into a wide character string while at the same time
converting all lowercase character into uppercase could look like this
(this is not the final version, just an example; it has no error
checking, and leaks sometimes memory):

     wchar_t *
     mbstouwcs (const char *s)
     {
       size_t len = strlen (s);
       wchar_t *result = malloc ((len + 1) * sizeof (wchar_t));
       wchar_t *wcp = result;
       wchar_t tmp[1];
       mbstate_t state;
       memset (&state, '\0', sizeof (state));
       size_t nbytes;
       while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0)
         {
           if (nbytes >= (size_t) -2)
             /* Invalid input string.  */
             return NULL;
           *result++ = towupper (tmp[0]);
           len -= nbytes;
           s += nbytes;
         }
       return result;
     }

   The use of `mbrtowc' should be clear.  A single wide character is
stored in `TMP[0]' and the number of consumed bytes is stored in the
variable NBYTES.  In case the the conversion was successful the
uppercase variant of the wide character is stored in the RESULT array
and the pointer to the input string and the number of available bytes
is adjusted.

   The only non-obvious thing about the function might be the way
memory is allocated for the result.  The above code uses the fact that
there can never be more wide characters in the converted results than
there are bytes in the multibyte input string.  This method yields to a
pessimistic guess about the size of the result and if many wide
character strings have to be constructed this way or the strings are
long, the extra memory required allocated because the input string
contains multibyte characters might be significant.  It would be
possible to resize the allocated memory block to the correct size before
returning it.  A better solution might be to allocate just the right
amount of space for the result right away.  Unfortunately there is no
function to compute the length of the wide character string directly
from the multibyte string.  But there is a function which does part of
the work.

 - Function: size_t mbrlen (const char *restrict S, size_t N, mbstate_t
          *PS)
     The `mbrlen' function ("multibyte restartable length") computes
     the number of at most N bytes starting at S which form the next
     valid and complete multibyte character.

     If the next multibyte character corresponds to the NUL wide
     character the return value is 0.  If the next N bytes form a valid
     multibyte character the number of bytes belonging to this multibyte
     character byte sequence is returned.

     If the the first N bytes possibly form a valid multibyte character
     but it is incomplete the return value is `(size_t) -2'.  Otherwise
     the multibyte character sequence is invalid and the return value
     is `(size_t) -1'.

     The multibyte sequence is interpreted in the state represented by
     the object pointed to by PS.  If PS is a null pointer, a state
     object local to `mbrlen' is used.

     This function was introduced in the second amendment to ISO C90 and
     is declared in `wchar.h'.

   The tentative reader now will of course note that `mbrlen' can be
implemented as

     mbrtowc (NULL, s, n, ps != NULL ? ps : &internal)

   This is true and in fact is mentioned in the official specification.
Now, how can this function be used to determine the length of the wide
character string created from a multibyte character string?  It is not
directly usable but we can define a function `mbslen' using it:

     size_t
     mbslen (const char *s)
     {
       mbstate_t state;
       size_t result = 0;
       size_t nbytes;
       memset (&state, '\0', sizeof (state));
       while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0)
         {
           if (nbytes >= (size_t) -2)
             /* Something is wrong.  */
             return (size_t) -1;
           s += nbytes;
           ++result;
         }
       return result;
     }

   This function simply calls `mbrlen' for each multibyte character in
the string and counts the number of function calls.  Please note that
we here use `MB_LEN_MAX' as the size argument in the `mbrlen' call.
This is OK since a) this value is larger then the length of the longest
multibyte character sequence and b) because we know that the string S
ends with a NUL byte which cannot be part of any other multibyte
character sequence but the one representing the NUL wide character.
Therefore the `mbrlen' function will never read invalid memory.

   Now that this function is available (just to make this clear, this
function is *not* part of the GNU C library) we can compute the number
of wide character required to store the converted multibyte character
string S using

     wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t);

   Please note that the `mbslen' function is quite inefficient.  The
implementation of `mbstouwcs' implemented using `mbslen' would have to
perform the conversion of the multibyte character input string twice
and this conversion might be quite expensive.  So it is necessary to
think about the consequences of using the easier but imprecise method
before doing the work twice.

 - Function: size_t wcrtomb (char *restrict S, wchar_t WC, mbstate_t
          *restrict PS)
     The `wcrtomb' function ("wide character restartable to multibyte")
     converts a single wide character into a multibyte string
     corresponding to that wide character.

     If S is a null pointer the function resets the the state stored in
     the objects pointer to by PS (or the internal `mbstate_t' object)
     to the initial state.  This can also be achieved by a call like
     this:

          wcrtombs (temp_buf, L'\0', ps)

     since if S is a null pointer `wcrtomb' performs as if it writes
     into an internal buffer which is guaranteed to be large enough.

     If WC is the NUL wide character `wcrtomb' emits, if necessary, a
     shift sequence to get the state PS into the initial state followed
     by a single NUL byte is stored in the string S.

     Otherwise a byte sequence (possibly including shift sequences) is
     written into the string S.  This of only happens if WC is a valid
     wide character, i.e., it has a multibyte representation in the
     character set selected by locale of the `LC_CTYPE' category.  If
     WC is no valid wide character nothing is stored in the strings S,
     `errno' is set to `EILSEQ', the conversion state in PS is
     undefined and the return value is `(size_t) -1'.

     If no error occurred the function returns the number of bytes
     stored in the string S.  This includes all byte representing shift
     sequences.

     One word about the interface of the function: there is no parameter
     specifying the length of the array S.  Instead the function
     assumes that there are at least `MB_CUR_MAX' bytes available since
     this is the maximum length of any byte sequence representing a
     single character.  So the caller has to make sure that there is
     enough space available, otherwise buffer overruns can occur.

     This function was introduced in the second amendment to ISO C and
     is declared in `wchar.h'.

   Using this function is as easy as using `mbrtowc'.  The following
example appends a wide character string to a multibyte character string.
Again, the code is not really useful (and correct), it is simply here to
demonstrate the use and some problems.

     char *
     mbscatwcs (char *s, size_t len, const wchar_t *ws)
     {
       mbstate_t state;
       /* Find the end of the existing string.  */
       char *wp = strchr (s, '\0');
       len -= wp - s;
       memset (&state, '\0', sizeof (state));
       do
         {
           size_t nbytes;
           if (len < MB_CUR_LEN)
             {
               /* We cannot guarantee that the next
                  character fits into the buffer, so
                  return an error.  */
               errno = E2BIG;
               return NULL;
             }
           nbytes = wcrtomb (wp, *ws, &state);
           if (nbytes == (size_t) -1)
             /* Error in the conversion.  */
             return NULL;
           len -= nbytes;
           wp += nbytes;
         }
       while (*ws++ != L'\0');
       return s;
     }

   First the function has to find the end of the string currently in the
array S.  The `strchr' call does this very efficiently since a
requirement for multibyte character representations is that the NUL byte
never is used except to represent itself (and in this context, the end
of the string).

   After initializing the state object the loop is entered where the
first task is to make sure there is enough room in the array S.  We
abort if there are not at least `MB_CUR_LEN' bytes available.  This is
not always optimal but we have no other choice.  We might have less
than `MB_CUR_LEN' bytes available but the next multibyte character
might also be only one byte long.  At the time the `wcrtomb' call
returns it is too late to decide whether the buffer was large enough or
not.  If this solution is really unsuitable there is a very slow but
more accurate solution.

       ...
       if (len < MB_CUR_LEN)
         {
           mbstate_t temp_state;
           memcpy (&temp_state, &state, sizeof (state));
           if (wcrtomb (NULL, *ws, &temp_state) > len)
             {
               /* We cannot guarantee that the next
                  character fits into the buffer, so
                  return an error.  */
               errno = E2BIG;
               return NULL;
             }
         }
       ...

   Here we do perform the conversion which might overflow the buffer so
that we are afterwards in the position to make an exact decision about
the buffer size.  Please note the `NULL' argument for the destination
buffer in the new `wcrtomb' call; since we are not interested in the
converted text at this point this is a nice way to express this.  The
most unusual thing about this piece of code certainly is the
duplication of the conversion state object.  But think about this: if a
change of the state is necessary to emit the next multibyte character
we want to have the same shift state change performed in the real
conversion.  Therefore we have to preserve the initial shift state
information.

   There are certainly many more and even better solutions to this
problem.  This example is only meant for educational purposes.


File: libc.info,  Node: Converting Strings,  Next: Multibyte Conversion Example,  Prev: Converting a Character,  Up: Restartable multibyte conversion

Converting Multibyte and Wide Character Strings
-----------------------------------------------

   The functions described in the previous section only convert a single
character at a time.  Most operations to be performed in real-world
programs include strings and therefore the ISO C standard also defines
conversions on entire strings.  However, the defined set of functions
is quite limited, thus the GNU C library contains a few extensions
which can help in some important situations.

 - Function: size_t mbsrtowcs (wchar_t *restrict DST, const char
          **restrict SRC, size_t LEN, mbstate_t *restrict PS)
     The `mbsrtowcs' function ("multibyte string restartable to wide
     character string") converts an NUL terminated multibyte character
     string at `*SRC' into an equivalent wide character string,
     including the NUL wide character at the end.  The conversion is
     started using the state information from the object pointed to by
     PS or from an internal object of `mbsrtowcs' if PS is a null
     pointer.  Before returning the state object to match the state
     after the last converted character.  The state is the initial
     state if the terminating NUL byte is reached and converted.

     If DST is not a null pointer the result is stored in the array
     pointed to by DST, otherwise the conversion result is not
     available since it is stored in an internal buffer.

     If LEN wide characters are stored in the array DST before reaching
     the end of the input string the conversion stops and LEN is
     returned.  If DST is a null pointer LEN is never checked.

     Another reason for a premature return from the function call is if
     the input string contains an invalid multibyte sequence.  In this
     case the global variable `errno' is set to `EILSEQ' and the
     function returns `(size_t) -1'.

     In all other cases the function returns the number of wide
     characters converted during this call.  If DST is not null
     `mbsrtowcs' stores in the pointer pointed to by SRC a null pointer
     (if the NUL byte in the input string was reached) or the address
     of the byte following the last converted multibyte character.

     This function was introduced in the second amendment to ISO C and
     is declared in `wchar.h'.

   The definition of this function has one limitation which has to be
understood.  The requirement that DST has to be a NUL terminated string
provides problems if one wants to convert buffers with text.  A buffer
is normally no collection of NUL terminated strings but instead a
continuous collection of lines, separated by newline characters.  Now
assume a function to convert one line from a buffer is needed.  Since
the line is not NUL terminated the source pointer cannot directly point
into the unmodified text buffer.  This means, either one inserts the NUL
byte at the appropriate place for the time of the `mbsrtowcs' function
call (which is not doable for a read-only buffer or in a multi-threaded
application) or one copies the line in an extra buffer where it can be
terminated by a NUL byte.  Note that it is not in general possible to
limit the number of characters to convert by setting the parameter LEN
to any specific value.  Since it is not known how many bytes each
multibyte character sequence is in length one always could do only a
guess.

   There is still a problem with the method of NUL-terminating a line
right after the newline character which could lead to very strange
results.  As said in the description of the MBSRTOWCS function above the
conversion state is guaranteed to be in the initial shift state after
processing the NUL byte at the end of the input string.  But this NUL
byte is not really part of the text.  I.e., the conversion state after
the newline in the original text could be something different than the
initial shift state and therefore the first character of the next line
is encoded using this state.  But the state in question is never
accessible to the user since the conversion stops after the NUL byte
(which resets the state).  Most stateful character sets in use today
require that the shift state after a newline is the initial state-but
this is not a strict guarantee.  Therefore simply NUL terminating a
piece of a running text is not always an adequate solution and therefore
never should be used in generally used code.

   The generic conversion interface (*note Generic Charset
Conversion::.)  does not have this limitation (it simply works on
buffers, not strings), and the GNU C library contains a set of
functions which take additional parameters specifying the maximal
number of bytes which are consumed from the input string.  This way the
problem of `mbsrtowcs''s example above could be solved by determining
the line length and passing this length to the function.

 - Function: size_t wcsrtombs (char *restrict DST, const wchar_t
          **restrict SRC, size_t LEN, mbstate_t *restrict PS)
     The `wcsrtombs' function ("wide character string restartable to
     multibyte string") converts the NUL terminated wide character
     string at `*SRC' into an equivalent multibyte character string and
     stores the result in the array pointed to by DST.  The NUL wide
     character is also converted.  The conversion starts in the state
     described in the object pointed to by PS or by a state object
     locally to `wcsrtombs' in case PS is a null pointer.  If DST is a
     null pointer the conversion is performed as usual but the result
     is not available.  If all characters of the input string were
     successfully converted and if DST is not a null pointer the
     pointer pointed to by SRC gets assigned a null pointer.

     If one of the wide characters in the input string has no valid
     multibyte character equivalent the conversion stops early, sets
     the global variable `errno' to `EILSEQ', and returns `(size_t) -1'.

     Another reason for a premature stop is if DST is not a null
     pointer and the next converted character would require more than
     LEN bytes in total to the array DST.  In this case (and if DEST is
     not a null pointer) the pointer pointed to by SRC is assigned a
     value pointing to the wide character right after the last one
     successfully converted.

     Except in the case of an encoding error the return value of the
     function is the number of bytes in all the multibyte character
     sequences stored in DST.  Before returning the state in the object
     pointed to by PS (or the internal object in case PS is a null
     pointer) is updated to reflect the state after the last
     conversion.  The state is the initial shift state in case the
     terminating NUL wide character was converted.

     This function was introduced in the second amendment to ISO C and
     is declared in `wchar.h'.

   The restriction mentions above for the `mbsrtowcs' function applies
also here.  There is no possibility to directly control the number of
input characters.  One has to place the NUL wide character at the
correct place or control the consumed input indirectly via the available
output array size (the LEN parameter).

 - Function: size_t mbsnrtowcs (wchar_t *restrict DST, const char
          **restrict SRC, size_t NMC, size_t LEN, mbstate_t *restrict
          PS)
     The `mbsnrtowcs' function is very similar to the `mbsrtowcs'
     function.  All the parameters are the same except for NMC which is
     new.  The return value is the same as for `mbsrtowcs'.

     This new parameter specifies how many bytes at most can be used
     from the multibyte character string.  I.e., the multibyte
     character string `*SRC' need not be NUL terminated.  But if a NUL
     byte is found within the NMC first bytes of the string the
     conversion stops here.

     This function is a GNU extensions.  It is meant to work around the
     problems mentioned above.  Now it is possible to convert buffer
     with multibyte character text piece for piece without having to
     care about inserting NUL bytes and the effect of NUL bytes on the
     conversion state.

   A function to convert a multibyte string into a wide character string
and display it could be written like this (this is not a really useful
example):

     void
     showmbs (const char *src, FILE *fp)
     {
       mbstate_t state;
       int cnt = 0;
       memset (&state, '\0', sizeof (state));
       while (1)
         {
           wchar_t linebuf[100];
           const char *endp = strchr (src, '\n');
           size_t n;
     
           /* Exit if there is no more line.  */
           if (endp == NULL)
             break;
     
           n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state);
           linebuf[n] = L'\0';
           fprintf (fp, "line %d: \"%S\"\n", linebuf);
         }
     }

   There is no problem with the state after a call to `mbsnrtowcs'.
Since we don't insert characters in the strings which were not in there
right from the beginning and we use STATE only for the conversion of
the given buffer there is no problem with altering the state.

 - Function: size_t wcsnrtombs (char *restrict DST, const wchar_t
          **restrict SRC, size_t NWC, size_t LEN, mbstate_t *restrict
          PS)
     The `wcsnrtombs' function implements the conversion from wide
     character strings to multibyte character strings.  It is similar to
     `wcsrtombs' but it takes, just like `mbsnrtowcs', an extra
     parameter which specifies the length of the input string.

     No more than NWC wide characters from the input string `*SRC' are
     converted.  If the input string contains a NUL wide character in
     the first NWC character to conversion stops at this place.

     This function is a GNU extension and just like `mbsnrtowcs' is
     helps in situations where no NUL terminated input strings are
     available.


File: libc.info,  Node: Multibyte Conversion Example,  Prev: Converting Strings,  Up: Restartable multibyte conversion

A Complete Multibyte Conversion Example
---------------------------------------

   The example programs given in the last sections are only brief and do
not contain all the error checking etc.  Presented here is a complete
and documented example.  It features the `mbrtowc' function but it
should be easy to derive versions using the other functions.

     int
     file_mbsrtowcs (int input, int output)
     {
       /* Note the use of `MB_LEN_MAX'.
          `MB_CUR_MAX' cannot portably be used here.  */
       char buffer[BUFSIZ + MB_LEN_MAX];
       mbstate_t state;
       int filled = 0;
       int eof = 0;
     
       /* Initialize the state.  */
       memset (&state, '\0', sizeof (state));
     
       while (!eof)
         {
           ssize_t nread;
           ssize_t nwrite;
           char *inp = buffer;
           wchar_t outbuf[BUFSIZ];
           wchar_t *outp = outbuf;
     
           /* Fill up the buffer from the input file.  */
           nread = read (input, buffer + filled, BUFSIZ);
           if (nread < 0)
             {
               perror ("read");
               return 0;
             }
           /* If we reach end of file, make a note to read no more. */
           if (nread == 0)
             eof = 1;
     
           /* `filled' is now the number of bytes in `buffer'. */
           filled += nread;
     
           /* Convert those bytes to wide characters-as many as we can. */
           while (1)
             {
               size_t thislen = mbrtowc (outp, inp, filled, &state);
               /* Stop converting at invalid character;
                  this can mean we have read just the first part
                  of a valid character.  */
               if (thislen == (size_t) -1)
                 break;
               /* We want to handle embedded NUL bytes
                  but the return value is 0.  Correct this.  */
               if (thislen == 0)
                 thislen = 1;
               /* Advance past this character. */
               inp += thislen;
               filled -= thislen;
               ++outp;
             }
     
           /* Write the wide characters we just made.  */
           nwrite = write (output, outbuf,
                           (outp - outbuf) * sizeof (wchar_t));
           if (nwrite < 0)
             {
               perror ("write");
               return 0;
             }
     
           /* See if we have a *real* invalid character. */
           if ((eof && filled > 0) || filled >= MB_CUR_MAX)
             {
               error (0, 0, "invalid multibyte character");
               return 0;
             }
     
           /* If any characters must be carried forward,
              put them at the beginning of `buffer'. */
           if (filled > 0)
             memmove (inp, buffer, filled);
         }
     
       return 1;
     }


File: libc.info,  Node: Non-reentrant Conversion,  Next: Generic Charset Conversion,  Prev: Restartable multibyte conversion,  Up: Character Set Handling

Non-reentrant Conversion Function
=================================

   The functions described in the last chapter are defined in the second
amendment to ISO C90.  But the original ISO C90 standard also contained
functions for character set conversion.  The reason that they are not
described in the first place is that they are almost entirely useless.

   The problem is that all the functions for conversion defined in
ISO C90 use a local state.  This implies that multiple conversions at
the same time (not only when using threads) cannot be done, and that you
cannot first convert single characters and then strings since you cannot
tell the conversion functions which state to use.

   These functions are therefore usable only in a very limited set of
situations.  One must complete converting the entire string before
starting a new one and each string/text must be converted with the same
function (there is no problem with the library itself; it is guaranteed
that no library function changes the state of any of these functions).
*For the above reasons it is highly requested that the functions from
the last section are used in place of non-reentrant conversion
functions.*

* Menu:

* Non-reentrant Character Conversion::  Non-reentrant Conversion of Single
                                         Characters.
* Non-reentrant String Conversion::     Non-reentrant Conversion of Strings.
* Shift State::                         States in Non-reentrant Functions.


File: libc.info,  Node: Non-reentrant Character Conversion,  Next: Non-reentrant String Conversion,  Up: Non-reentrant Conversion

Non-reentrant Conversion of Single Characters
---------------------------------------------

 - Function: int mbtowc (wchar_t *restrict RESULT, const char *restrict
          STRING, size_t SIZE)
     The `mbtowc' ("multibyte to wide character") function when called
     with non-null STRING converts the first multibyte character
     beginning at STRING to its corresponding wide character code.  It
     stores the result in `*RESULT'.

     `mbtowc' never examines more than SIZE bytes.  (The idea is to
     supply for SIZE the number of bytes of data you have in hand.)

     `mbtowc' with non-null STRING distinguishes three possibilities:
     the first SIZE bytes at STRING start with valid multibyte
     character, they start with an invalid byte sequence or just part
     of a character, or STRING points to an empty string (a null
     character).

     For a valid multibyte character, `mbtowc' converts it to a wide
     character and stores that in `*RESULT', and returns the number of
     bytes in that character (always at least 1, and never more than
     SIZE).

     For an invalid byte sequence, `mbtowc' returns -1.  For an empty
     string, it returns 0, also storing `'\0'' in `*RESULT'.

     If the multibyte character code uses shift characters, then
     `mbtowc' maintains and updates a shift state as it scans.  If you
     call `mbtowc' with a null pointer for STRING, that initializes the
     shift state to its standard initial value.  It also returns
     nonzero if the multibyte character code in use actually has a
     shift state.  *Note Shift State::.

 - Function: int wctomb (char *STRING, wchar_t WCHAR)
     The `wctomb' ("wide character to multibyte") function converts the
     wide character code WCHAR to its corresponding multibyte character
     sequence, and stores the result in bytes starting at STRING.  At
     most `MB_CUR_MAX' characters are stored.

     `wctomb' with non-null STRING distinguishes three possibilities
     for WCHAR: a valid wide character code (one that can be translated
     to a multibyte character), an invalid code, and `L'\0''.

     Given a valid code, `wctomb' converts it to a multibyte character,
     storing the bytes starting at STRING.  Then it returns the number
     of bytes in that character (always at least 1, and never more than
     `MB_CUR_MAX').

     If WCHAR is an invalid wide character code, `wctomb' returns -1.
     If WCHAR is `L'\0'', it returns `0', also storing `'\0'' in
     `*STRING'.

     If the multibyte character code uses shift characters, then
     `wctomb' maintains and updates a shift state as it scans.  If you
     call `wctomb' with a null pointer for STRING, that initializes the
     shift state to its standard initial value.  It also returns
     nonzero if the multibyte character code in use actually has a
     shift state.  *Note Shift State::.

     Calling this function with a WCHAR argument of zero when STRING is
     not null has the side-effect of reinitializing the stored shift
     state *as well as* storing the multibyte character `'\0'' and
     returning 0.

   Similar to `mbrlen' there is also a non-reentrant function which
computes the length of a multibyte character.  It can be defined in
terms of `mbtowc'.

 - Function: int mblen (const char *STRING, size_t SIZE)
     The `mblen' function with a non-null STRING argument returns the
     number of bytes that make up the multibyte character beginning at
     STRING, never examining more than SIZE bytes.  (The idea is to
     supply for SIZE the number of bytes of data you have in hand.)

     The return value of `mblen' distinguishes three possibilities: the
     first SIZE bytes at STRING start with valid multibyte character,
     they start with an invalid byte sequence or just part of a
     character, or STRING points to an empty string (a null character).

     For a valid multibyte character, `mblen' returns the number of
     bytes in that character (always at least `1', and never more than
     SIZE).  For an invalid byte sequence, `mblen' returns -1.  For an
     empty string, it returns 0.

     If the multibyte character code uses shift characters, then `mblen'
     maintains and updates a shift state as it scans.  If you call
     `mblen' with a null pointer for STRING, that initializes the shift
     state to its standard initial value.  It also returns a nonzero
     value if the multibyte character code in use actually has a shift
     state.  *Note Shift State::.

     The function `mblen' is declared in `stdlib.h'.


File: libc.info,  Node: Non-reentrant String Conversion,  Next: Shift State,  Prev: Non-reentrant Character Conversion,  Up: Non-reentrant Conversion

Non-reentrant Conversion of Strings
-----------------------------------

   For convenience reasons the ISO C90 standard defines also functions
to convert entire strings instead of single characters.  These functions
suffer from the same problems as their reentrant counterparts from the
second amendment to ISO C90; see *Note Converting Strings::.

 - Function: size_t mbstowcs (wchar_t *WSTRING, const char *STRING,
          size_t SIZE)
     The `mbstowcs' ("multibyte string to wide character string")
     function converts the null-terminated string of multibyte
     characters STRING to an array of wide character codes, storing not
     more than SIZE wide characters into the array beginning at WSTRING.
     The terminating null character counts towards the size, so if SIZE
     is less than the actual number of wide characters resulting from
     STRING, no terminating null character is stored.

     The conversion of characters from STRING begins in the initial
     shift state.

     If an invalid multibyte character sequence is found, this function
     returns a value of -1.  Otherwise, it returns the number of wide
     characters stored in the array WSTRING.  This number does not
     include the terminating null character, which is present if the
     number is less than SIZE.

     Here is an example showing how to convert a string of multibyte
     characters, allocating enough space for the result.

          wchar_t *
          mbstowcs_alloc (const char *string)
          {
            size_t size = strlen (string) + 1;
            wchar_t *buf = xmalloc (size * sizeof (wchar_t));
          
            size = mbstowcs (buf, string, size);
            if (size == (size_t) -1)
              return NULL;
            buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
            return buf;
          }


 - Function: size_t wcstombs (char *STRING, const wchar_t *WSTRING,
          size_t SIZE)
     The `wcstombs' ("wide character string to multibyte string")
     function converts the null-terminated wide character array WSTRING
     into a string containing multibyte characters, storing not more
     than SIZE bytes starting at STRING, followed by a terminating null
     character if there is room.  The conversion of characters begins in
     the initial shift state.

     The terminating null character counts towards the size, so if SIZE
     is less than or equal to the number of bytes needed in WSTRING, no
     terminating null character is stored.

     If a code that does not correspond to a valid multibyte character
     is found, this function returns a value of -1.  Otherwise, the
     return value is the number of bytes stored in the array STRING.
     This number does not include the terminating null character, which
     is present if the number is less than SIZE.


File: libc.info,  Node: Shift State,  Prev: Non-reentrant String Conversion,  Up: Non-reentrant Conversion

States in Non-reentrant Functions
---------------------------------

   In some multibyte character codes, the *meaning* of any particular
byte sequence is not fixed; it depends on what other sequences have come
earlier in the same string.  Typically there are just a few sequences
that can change the meaning of other sequences; these few are called
"shift sequences" and we say that they set the "shift state" for other
sequences that follow.

   To illustrate shift state and shift sequences, suppose we decide that
the sequence `0200' (just one byte) enters Japanese mode, in which
pairs of bytes in the range from `0240' to `0377' are single
characters, while `0201' enters Latin-1 mode, in which single bytes in
the range from `0240' to `0377' are characters, and interpreted
according to the ISO Latin-1 character set.  This is a multibyte code
which has two alternative shift states ("Japanese mode" and "Latin-1
mode"), and two shift sequences that specify particular shift states.

   When the multibyte character code in use has shift states, then
`mblen', `mbtowc' and `wctomb' must maintain and update the current
shift state as they scan the string.  To make this work properly, you
must follow these rules:

   * Before starting to scan a string, call the function with a null
     pointer for the multibyte character address--for example, `mblen
     (NULL, 0)'.  This initializes the shift state to its standard
     initial value.

   * Scan the string one character at a time, in order.  Do not "back
     up" and rescan characters already scanned, and do not intersperse
     the processing of different strings.

   Here is an example of using `mblen' following these rules:

     void
     scan_string (char *s)
     {
       int length = strlen (s);
     
       /* Initialize shift state.  */
       mblen (NULL, 0);
     
       while (1)
         {
           int thischar = mblen (s, length);
           /* Deal with end of string and invalid characters.  */
           if (thischar == 0)
             break;
           if (thischar == -1)
             {
               error ("invalid multibyte character");
               break;
             }
           /* Advance past this character.  */
           s += thischar;
           length -= thischar;
         }
     }

   The functions `mblen', `mbtowc' and `wctomb' are not reentrant when
using a multibyte code that uses a shift state.  However, no other
library functions call these functions, so you don't have to worry that
the shift state will be changed mysteriously.


File: libc.info,  Node: Generic Charset Conversion,  Prev: Non-reentrant Conversion,  Up: Character Set Handling

Generic Charset Conversion
==========================

   The conversion functions mentioned so far in this chapter all had in
common that they operate on character sets which are not directly
specified by the functions.  The multibyte encoding used is specified by
the currently selected locale for the `LC_CTYPE' category.  The wide
character set is fixed by the implementation (in the case of GNU C
library it always is UCS4 encoded ISO 10646.

   This has of course several problems when it comes to general
character conversion:

   * For every conversion where neither the source or destination
     character set is the character set of the locale for the
     `LC_CTYPE' category, one has to change the `LC_CTYPE' locale using
     `setlocale'.

     This introduces major problems for the rest of the programs since
     several more functions (e.g., the character classification
     functions, *note Classification of Characters::.) use the
     `LC_CTYPE' category.

   * Parallel conversions to and from different character sets are not
     possible since the `LC_CTYPE' selection is global and shared by all
     threads.

   * If neither the source nor the destination character set is the
     character set used for `wchar_t' representation there is at least
     a two-step process necessary to convert a text using the functions
     above.  One would have to select the source character set as the
     multibyte encoding, convert the text into a `wchar_t' text, select
     the destination character set as the multibyte encoding and
     convert the wide character text to the multibyte (= destination)
     character set.

     Even if this is possible (which is not guaranteed) it is a very
     tiring work.  Plus it suffers from the other two raised points
     even more due to the steady changing of the locale.

   The XPG2 standard defines a completely new set of functions which has
none of these limitations.  They are not at all coupled to the selected
locales and they but no constraints on the character sets selected for
source and destination.  Only the set of available conversions is
limiting them.  The standard does not specify that any conversion at all
must be available.  It is a measure of the quality of the
implementation.

   In the following text first the interface to `iconv', the conversion
function, will be described.  Comparisons with other implementations
will show what pitfalls lie on the way of portable applications.  At
last, the implementation is described as far as interesting to the
advanced user who wants to extend the conversion capabilities.

* Menu:

* Generic Conversion Interface::    Generic Character Set Conversion Interface.
* iconv Examples::                  A complete `iconv' example.
* Other iconv Implementations::     Some Details about other `iconv'
                                     Implementations.
* glibc iconv Implementation::      The `iconv' Implementation in the GNU C
                                     library.

