libunibreak: src/linebreak.c File Reference

# src/linebreak.c File Reference

Implementation of the line breaking algorithm as described in Unicode Standard Annex 14. More...

`#include <assert.h>`
`#include <stddef.h>`
`#include <string.h>`
`#include "linebreak.h"`
`#include "linebreakdef.h"`
Include dependency graph for linebreak.c:

## Data Structures
struct | LineBreakPropertiesIndex
| Struct for the second-level index to the line breaking properties. More...
## Defines
#define | LINEBREAK_INDEX_SIZE 40
| Size of the second-level index to the line breaking properties.
## Enumerations
enum | BreakAction { DIR_BRK, IND_BRK, CMI_BRK, CMP_BRK, PRH_BRK }
| Enumeration of break actions. More...
## Functions
void | init_linebreak (void)
| Initializes the second-level index to the line breaking properties.
static struct LineBreakProperties * | get_lb_prop_lang (const char *lang)
| Gets the language-specific line breaking properties.
static enum LineBreakClass | get_char_lb_class (utf32_t ch, struct LineBreakProperties *lbp)
| Gets the line breaking class of a character from a line breaking properties array.
static enum LineBreakClass | get_char_lb_class_default (utf32_t ch)
| Gets the line breaking class of a character from the default line breaking properties array.
static enum LineBreakClass | get_char_lb_class_lang (utf32_t ch, struct LineBreakProperties *lbpLang)
| Gets the line breaking class of a character for a specific language.
static enum LineBreakClass | resolve_lb_class (enum LineBreakClass lbc, const char *lang)
| Resolves the line breaking class for certain ambiguous or complicated characters.
utf32_t | lb_get_next_char_utf8 (const utf8_t *s, size_t len, size_t *ip)
| Gets the next Unicode character in a UTF-8 sequence.
utf32_t | lb_get_next_char_utf16 (const utf16_t *s, size_t len, size_t *ip)
| Gets the next Unicode character in a UTF-16 sequence.
utf32_t | lb_get_next_char_utf32 (const utf32_t *s, size_t len, size_t *ip)
| Gets the next Unicode character in a UTF-32 sequence.
void | set_linebreaks (const void *s, size_t len, const char *lang, char *brks, get_next_char_t get_next_char)
| Sets the line breaking information for a generic input string.
void | set_linebreaks_utf8 (const utf8_t *s, size_t len, const char *lang, char *brks)
| Sets the line breaking information for a UTF-8 input string.
void | set_linebreaks_utf16 (const utf16_t *s, size_t len, const char *lang, char *brks)
| Sets the line breaking information for a UTF-16 input string.
void | set_linebreaks_utf32 (const utf32_t *s, size_t len, const char *lang, char *brks)
| Sets the line breaking information for a UTF-32 input string.
int | is_line_breakable (utf32_t char1, utf32_t char2, const char *lang)
| Tells whether a line break can occur between two Unicode characters.
## Variables
const int | linebreak_version = LINEBREAK_VERSION
| Version number of the library.
static enum BreakAction | baTable [LBP_RI][LBP_RI]
| Break action pair table.
static struct LineBreakPropertiesIndex | lb_prop_index [LINEBREAK_INDEX_SIZE]
| Second-level index to the line breaking properties.

## Detailed Description

Implementation of the line breaking algorithm as described in Unicode Standard Annex 14.

Version:
  2.3, 2012/10/06

Author:
  Wu Yongwei

## Define Documentation

#define LINEBREAK_INDEX_SIZE 40

Size of the second-level index to the line breaking properties.

## Enumeration Type Documentation

enum BreakAction

Enumeration of break actions.

They are used in the break action pair table below.

Enumerator:

  DIR_BRK | Direct break opportunity.
  IND_BRK | Indirect break opportunity.
  CMI_BRK | Indirect break opportunity for combining marks.
  CMP_BRK | Prohibited break for combining marks.
  PRH_BRK | Prohibited break.

## Function Documentation

static enum LineBreakClass get_char_lb_class | ( | utf32_t | ch,
|  | struct LineBreakProperties * | lbp |
| ) |  |  | `[static]`

Gets the line breaking class of a character from a line breaking properties array.

Parameters:

  | ch | character to check
  | lbp | pointer to the line breaking properties array

Returns:
  the line breaking class if found; `LBP_XX` otherwise

static enum LineBreakClass get_char_lb_class_default | ( | utf32_t | ch | ) | `[static]`

Gets the line breaking class of a character from the default line breaking properties array.

Parameters:

  | ch | character to check

Returns:
  the line breaking class if found; `LBP_XX` otherwise

static enum LineBreakClass get_char_lb_class_lang | ( | utf32_t | ch,
|  | struct LineBreakProperties * | lbpLang |
| ) |  |  | `[static]`

Gets the line breaking class of a character for a specific language.

This function will check the language-specific data first, and then the default data if there is no language-specific property available for the character.

Parameters:

  | ch | character to check
  | lbpLang | pointer to the language-specific line breaking properties array

Returns:
  the line breaking class if found; `LBP_XX` otherwise

static struct LineBreakProperties* get_lb_prop_lang | ( | const char * | lang | ) | `[static, read]`

Gets the language-specific line breaking properties.

Parameters:

  | lang | language of the text

Returns:
  pointer to the language-specific line breaking properties array if found; `NULL` otherwise

void init_linebreak | ( | void |  | ) |

Initializes the second-level index to the line breaking properties.

If it is not called, the performance of get_char_lb_class_lang (and thus the main functionality) can be pretty bad, especially for big code points like those of Chinese.

int is_line_breakable | ( | utf32_t | char1,
|  | utf32_t | char2,
|  | const char * | lang |
| ) |  |  |

Tells whether a line break can occur between two Unicode characters.

This is a wrapper function to expose a simple interface. Generally speaking, it is better to use set_linebreaks_utf32 instead, since complicated cases involving combining marks, spaces, etc. cannot be correctly processed.

Parameters:

  | char1 | the first Unicode character
  | char2 | the second Unicode character
  | lang | language of the input

Returns:
  one of LINEBREAK_MUSTBREAK, LINEBREAK_ALLOWBREAK, LINEBREAK_NOBREAK, or LINEBREAK_INSIDEACHAR

utf32_t lb_get_next_char_utf16 | ( | const utf16_t * | s,
|  | size_t | len,
|  | size_t * | ip |
| ) |  |  |

Gets the next Unicode character in a UTF-16 sequence.

The index will be advanced to the next complete character, unless the end of string is reached in the middle of a UTF-16 surrogate pair.

Parameters:

  `[in]` | s | input UTF-16 string
  `[in]` | len | length of the string in words
  `[in,out]` | ip | pointer to the index

Returns:
  the Unicode character beginning at the index; or EOS if end of input is encountered

utf32_t lb_get_next_char_utf32 | ( | const utf32_t * | s,
|  | size_t | len,
|  | size_t * | ip |
| ) |  |  |

Gets the next Unicode character in a UTF-32 sequence.

The index will be advanced to the next character.

Parameters:

  `[in]` | s | input UTF-32 string
  `[in]` | len | length of the string in dwords
  `[in,out]` | ip | pointer to the index

Returns:
  the Unicode character beginning at the index; or EOS if end of input is encountered

utf32_t lb_get_next_char_utf8 | ( | const utf8_t * | s,
|  | size_t | len,
|  | size_t * | ip |
| ) |  |  |

Gets the next Unicode character in a UTF-8 sequence.

The index will be advanced to the next complete character, unless the end of string is reached in the middle of a UTF-8 sequence.

Parameters:

  `[in]` | s | input UTF-8 string
  `[in]` | len | length of the string in bytes
  `[in,out]` | ip | pointer to the index

Returns:
  the Unicode character beginning at the index; or EOS if end of input is encountered

static enum LineBreakClass resolve_lb_class | ( | enum LineBreakClass | lbc,
|  | const char * | lang |
| ) |  |  | `[static]`

Resolves the line breaking class for certain ambiguous or complicated characters.

They are treated in a simplistic way in this implementation.

Parameters:

  | lbc | line breaking class to resolve
  | lang | language of the text

Returns:
  the resolved line breaking class

void set_linebreaks | ( | const void * | s,
|  | size_t | len,
|  | const char * | lang,
|  | char * | brks,
|  | get_next_char_t | get_next_char |
| ) |  |  |

Sets the line breaking information for a generic input string.

Parameters:

  `[in]` | s | input string
  `[in]` | len | length of the input
  `[in]` | lang | language of the input
  `[out]` | brks | pointer to the output breaking data, containing LINEBREAK_MUSTBREAK, LINEBREAK_ALLOWBREAK, LINEBREAK_NOBREAK, or LINEBREAK_INSIDEACHAR
  `[in]` | get_next_char | function to get the next UTF-32 character

void set_linebreaks_utf16 | ( | const utf16_t * | s,
|  | size_t | len,
|  | const char * | lang,
|  | char * | brks |
| ) |  |  |

Sets the line breaking information for a UTF-16 input string.

Parameters:

  `[in]` | s | input UTF-16 string
  `[in]` | len | length of the input
  `[in]` | lang | language of the input
  `[out]` | brks | pointer to the output breaking data, containing LINEBREAK_MUSTBREAK, LINEBREAK_ALLOWBREAK, LINEBREAK_NOBREAK, or LINEBREAK_INSIDEACHAR

void set_linebreaks_utf32 | ( | const utf32_t * | s,
|  | size_t | len,
|  | const char * | lang,
|  | char * | brks |
| ) |  |  |

Sets the line breaking information for a UTF-32 input string.

Parameters:

  `[in]` | s | input UTF-32 string
  `[in]` | len | length of the input
  `[in]` | lang | language of the input
  `[out]` | brks | pointer to the output breaking data, containing LINEBREAK_MUSTBREAK, LINEBREAK_ALLOWBREAK, LINEBREAK_NOBREAK, or LINEBREAK_INSIDEACHAR

void set_linebreaks_utf8 | ( | const utf8_t * | s,
|  | size_t | len,
|  | const char * | lang,
|  | char * | brks |
| ) |  |  |

Sets the line breaking information for a UTF-8 input string.

Parameters:

  `[in]` | s | input UTF-8 string
  `[in]` | len | length of the input
  `[in]` | lang | language of the input
  `[out]` | brks | pointer to the output breaking data, containing LINEBREAK_MUSTBREAK, LINEBREAK_ALLOWBREAK, LINEBREAK_NOBREAK, or LINEBREAK_INSIDEACHAR

## Variable Documentation

enum BreakAction baTable[LBP_RI][LBP_RI]`[static]`

Break action pair table.

This is a direct mapping of Table 2 of Unicode Standard Annex 14, Revision 30.

struct LineBreakPropertiesIndex lb_prop_index[LINEBREAK_INDEX_SIZE]`[static]`

Initial value:

    {
        { 0xFFFFFFFF, lb_prop_default }
    }

Second-level index to the line breaking properties.

const int linebreak_version = LINEBREAK_VERSION

Version number of the library.