
=head1 NAME

ShiftJIS::Collate - collation of Shift-JIS strings

=head1 SYNOPSIS

  use ShiftJIS::Collate;

  @sorted = ShiftJIS::Collate->new(%tailoring)->sort(@source);

=head1 ABOUT THIS POD

This POD is written in Shift-JIS.

Do you see 'C<あ>' as C<HIRAGANA LETTER A>?
or 'C<\>' as C<YEN SIGN>, not as C<REVERSE SOLIDUS>?
Otherwise you'd change your font to an appropriate one.
(or the POD might be badly converted.)

=head1 DESCRIPTION

This module provides some functions to compare and sort strings
in Shift-JIS based on B<JIS X 4061:1996>,
collation of Japanese character strings,
"Nihongo mojiretsu shogo junban", 「日本語文字列照合順番」.

This module is an implementation of JIS X 4061:1996 and
the collation rules are based on that standard.
See L<Conformance to the Standard>.

=head2 Constructor and Tailoring

The C<new> method returns a collator object.

   $Collator = ShiftJIS::Collate->new(
      ignoreChar => $regexIgnoredChar,
      kanji => $kanji_class,
      katakana_before_hiragana => $bool,
      level => $collationLevel,
      position_in_bytes => $bool,
      tounicode  => \&sjis_to_unicode,
      preprocess => \&preprocess,
      upper_before_lower => $bool,
   );
   # if %tailoring is false (empty),
   # $Collator should do the default collation.

=over 4

=item ignoreChar

If specified as a regular expression,
any characters that match it are ignored on collation.

e.g. If you want to ignore C<KATAKANA-HIRAGANA PROLONGED SOUND MARK>
and its halfwidth form, say

   ignoreChar => '^(?:\x81\x5B|\xB0)',

=item katakana_before_hiragana

By default, hiragana is before katakana.

If the parameter is true, this is reversed.

=item kanji

Set the kanji class. See L<Kanji Classes>.

  Class 1: 'saisho' (minimal)
  Class 2: 'kihon' (basic)
  Class 3: 'kakucho' (extended)

The kanji class is specified as 1, 2, or 3. If omitted, class 2 is applied.

This module does not support 'kakucho' kanji class,
since the repertoire of Shift-JIS does not define
all the Unicode CJK unified ideographs.

But if the kanji class 3 is specified, you can collate kanji
in the unicode order. In this case you must provide
L<tounicode> coderef which gives a unicode codepoint
from a Shift-JIS character.

=item level

Set the maximum level. See L<Collation Levels>.
Any higher levels than the specified one are ignored.

  Level 1: alphabetic ordering
  Level 2: diacritic ordering
  Level 3: case ordering
  Level 4: script ordering
  Level 5: width ordering

The collation level is specified as a number
between 1 and 5. If omitted, level 4 is applied.

=item tounicode

If you want to collate kanji in the unicode order,
specify a coderef which gives a unicode codepoint
from a Shift-JIS character.

Such a subroutine should map a string comprising of
a kanji of level 1 and 2 in Shift-JIS to a codepoint
in the range between 0x4E00 and 0x9FFF.

=item position_in_bytes

By default, the C<index> method returns its results in characters.

If this parameter is true, it returns the results in bytes.

=item preprocess

If specified, the coderef is used to preprocess
before the formation of sort keys.
I.e. a sort key is formed from a return value returned by the coderef.

=item upper_before_lower

By default, lowercase is before uppercase.

If the parameter is true, this is reversed.

=back

=head2 Comparison

=over 4

=item C<$result = $Collator-E<gt>cmp($a, $b)>

Returns 1 (when C<$a> is greater than C<$b>)
or 0 (when C<$a> is equal to C<$b>)
or -1 (when C<$a> is lesser than C<$b>).

=item C<$result = $Collator-E<gt>eq($a, $b)>

=item C<$result = $Collator-E<gt>ne($a, $b)>

=item C<$result = $Collator-E<gt>gt($a, $b)>

=item C<$result = $Collator-E<gt>ge($a, $b)>

=item C<$result = $Collator-E<gt>lt($a, $b)>

=item C<$result = $Collator-E<gt>le($a, $b)>

They works like the same name operators as theirs.

=back

=head2 Sorting

=over 4

=item C<$sortKey = $Collator-E<gt>getSortKey($string)>

Returns a sort key.

You compare the sort keys using a binary comparison
and get the result of the comparison of the strings.

   $Collator->getSortKey($a) cmp $Collator->getSortKey($b)

      is equivalent to

   $Collator->cmp($a, $b)

=item C<@sorted = $Collator-E<gt>sort(@source)>

Sorts a list of strings by B<tanjun shogo>: 'the simple collation'.

=item C<@sorted = $Collator-E<gt>sortYomi(@source)>

Sorts a list of references to arrays of (spell, reading)
by B<yomi-hyoki shogo>: 'the collation using readings and spells'.

E.g., an element of @source is probably C<['日本語', 'にほんご']>;
Though its opposite, C<['にほんご', '日本語']>, is also allowed,
the order of (spell, reading) is not varied.

B<Yomi-hyoki shogo> is carried out through two comparison stages.

E.g., sort these strings by 'Yomi-hyoki shogo'.

C<['永田', 'ながた']>, C<['小山', 'おやま']>, C<['長田', 'おさだ']>,
C<['長田', 'ながた']>, C<['小山', 'こやま']>.

First, order by reading
(C<'おさだ'> E<lt> C<'おやま'> E<lt> 'こやま' E<lt> 'ながた');
next, order by spelling among strings having the same reading
(C<'永田'> E<lt> C<'長田'> where both are read as C<'ながた'>).

The result should be C<['長田', 'おさだ']> E<lt> C<['小山', 'おやま']>
E<lt> C<['小山', 'こやま']> E<lt> C<['永田', 'ながた']> E<lt>
C<['長田', 'ながた']>.

See also F<sample/yomi.txt>.

=item C<@sorted = $Collator-E<gt>sortDaihyo(@source)>

Sorts a list of references to arrays of (spell, reading)
by B<kan'i-daihyo-yomi shogo>:
'the simplified representative reading collation'.

B<Note>: This module does not support B<kihon-daihyo-yomi shogo>:
'the basic representative reading collation',
in which B<daihyo-yomi jisho'>, a dictionary of representative readings,
is required.

B<kan'i-daihyo-yomi shogo> is carried out through B<five> comparison stages.
This ordered list is an example of the result of C<"kan'i-daihyo-yomi shogo">.

  ['４面体', 'しめんたい'],
  ['２色性', 'にしょくせい'],
  ['４次元', 'よじげん'],
  ['６面体', 'ろくめんたい'],
  ['α崩壊', 'アルファほうかい'],
  ['Γ関数', 'ガンマかんすう'],
  ['β線',   'ベータせん'],
  ['Ｑ値',   'キューち'],
  ['ＪＩＳ', 'じす'],
  ['Perl',   'パール'],
  ['河西',   'かさい'],
  ['河合',   'かわい'],
  ['河田',   'かわだ'],
  ['河内',   'かわち'],
  ['河辺',   'かわべ'],
  ['角田',   'かくた'],
  ['角田',   'かどた'],
  ['関東',   'かんとう'],
  ['河内',   'こうち'],
  ['沢島',   'さわしま'],
  ['沢嶋',   'さわしま'],
  ['沢田',   'さわだ'],
  ['澤島',   'さわしま'],
  ['澤嶋',   'さわしま'],
  ['澤田',   'さわだ'],
  ['角田',   'つのだ'],
  ['土井',   'つちい'],
  ['土居',   'つちい'],
  ['土井',   'どい'],
  ['土居',   'どい'],

(1) Compare the character class of the first character of the spell.

  Digit class ('４面体') < Greek class ('α崩壊') <
Latin class ('ＪＩＳ') < Kanji class ('関東').

(2) Compare the first character of the reading.

  e.g. 'しめんたい' < 'にしょくせい' < 'よじげん' < 'ろくめんたい'.

(3) Compare the first character of the spell.

  e.g. ('河西','河田',etc.) < ('角田','角田') < ('関東');

       ('沢島','沢嶋','沢田') < ('澤島','澤嶋','澤田').

(4) Compare the whole string of the reading.

  e.g. ['河西', 'かさい'] < ['河合', 'かわい'] < ['河田', 'かわだ'];

       ['角田', 'かくた'] < ['角田', 'かどた'].

(5) Compare the whole string of the spell.

  e.g. ['沢島', 'さわしま'] < ['沢嶋', 'さわしま'] < ['沢田', 'さわだ'].

See also F<sample/daihyo.txt>.

=back

=head2 Searching

=over 4

=item C<$position = $Collator-E<gt>index($string, $substring)>

=item C<($position, $length) = $Collator-E<gt>index($string, $substring)>

If C<$substring> matches a part of C<$string>, returns
the position of the first occurrence of the matching part in scalar context;
in list context, returns a two-element list of
the position and the length of the matching part.

B<Notice> that the length of the matching part may differ from
the length of C<$substring>.

If C<$substring> does not match any part of C<$string>,
returns C<-1> in scalar context and
an empty list in list context.

e.g. you say

  use ShiftJIS::Collate;
  use ShiftJIS::String qw(substr);

  my $Col = ShiftJIS::Collate->new( level => $level );
  my $str = "* ひらがなとカタカナはレベル３では等しいかな。";
  my $sub = "かな";
  my $match;
  if (my @tmp = $Col->index($str, $sub)) {
    $match = substr($str, $tmp[0], $tmp[1]);
  }

If C<$level> is 1, you get C<"がな">;
if C<$level> is 2 or 3, you get C<"カナ">;
if C<$level> is 4 or 5, you get C<"かな">.

If your C<substr> function is aware of Shift-JIS,
specify true as C<position_in_bytes>. See L<Constructor and Tailoring>.

=back

=head1 NOTE

=head2 Collation Levels

The following criteria are considered in order
until the collation order is determined.
By default, Levels 1 to 4 are applied and Level 5 is ignored,
for JIS X 4061 does not specify level 5.

=over 4

=item Level 1: alphabetic ordering.

Character classes ordered as following:

    1    space
    2   'kijutsu kigou' (punctuation marks)
    3   'kakko kigou' (quotes, parentheses, and braces)
    4   'gakujutsu kigou' (mathematical and technical signs)
    5   'ippan kigou' (general symbols)
    6   'tan-i kigou' (unit signs)
    7    digits (European)
    8   'ouji kigou' (Greek and Cyrillic letters, as "European signs")
    9    Latin alphabets
   10    kana (haragana and katakana)
   11    kanji
   12   'geta' mark

In the class, alphabets are collated alphabetically;
kana letters are AIUEO-betically (in the Gozyuon order, '五十音順');
kanji are in the JIS X 0208 order.

According to JIS X 4061, C<GETA MARK> ('〓', 0x81AC, U+3013)
must be the greatest character (ordered at the last).

Any character for which the order is no defined,
like control characters, box drawings, unassigned characters, etc.
is regarded as a completely ignorable character.

=item Level 2: diacritic ordering.

In kana, the order is as shown the following list.

    A voiceless kana, the voiced, then the semi-voiced (if exists).
     (eg. 'か' < 'が'; 'は' < 'ば' < 'ぱ')

=item Level 3: case ordering.

A small Latin is lesser than the corresponding Capital.

In kana, the order is as shown the following list.
see L<Replacement of PROLONGED SOUND MARK and ITERATION MARKs>.

    Replaced PROLONGED SOUND MARKs (U+30FC and U+FF70);
    Small Kana;
    Replaced ITERATION MARKs (U+309D, U+309E, U+30FD, and U+30FE);
    then, Normal Kana.

    Then, e.g., 'あー' < 'あぁ' < 'あゝ' < 'ああ'.

=item Level 4: script ordering.

Any hiragana is lesser than the corresponding katakana.

    Then, e.g., 'あ' < 'ア'.

=item Level 5: width ordering.

A character that belongs to the block
C<Halfwidth and Fullwidth Forms>
is greater than the corresponding normal character.

B<BN: JIS X 4061 does not mention this level.
Level 5 is an extention by this module.>

=back

=head2 Kanji Classes

JIS X 4061 specifies three Kanji Classes.
This modules supports 'Saisho' and 'Kihon' Kanji Classes.

=over 4

=item the 'saisho' (minimal) kanji class

It comprises five kanji-like characters:

    '〃' (0x8156, U+3003)
    '々' (0x8158, U+3005)
    '仝' (0x8157, U+4EDD)
    '〆' (0x8159, U+3006)
    '〇' (0x815A, U+3007)

Any kanji except '仝' are ignored on collation.

=item the 'kihon' (basic) kanji class

It comprises JIS level 1 and 2 kanji in addition to
the minimal kanji class. Sorted in the JIS codepoint order.
Any kanji excepting those defined by JIS X 0208 are ignored on collation.

=item the 'kakucho' (extended) kanji class

All the CJK Unified Ideographs in addition to
the minimal kanji class. Sorted in the Unicode codepoint order.

=back

=head2 Replacement of PROLONGED SOUND MARKs and ITERATION MARKs

   Character  SJIS    UCS     Name

     'ー'    0x815B  U+30FC  KATAKANA-HIRAGANA PROLONGED SOUND MARK
     'ｰ'     0xB0    U+FF70  HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK
     'ゝ'    0x8154  U+309D  HIRAGANA ITERATION MARK
     'ゞ'    0x8155  U+309E  HIRAGANA VOICED ITERATION MARK
     'ヽ'    0x8152  U+30FD  KATAKANA ITERATION MARK
     'ヾ'    0x8153  U+30FE  KATAKANA VOICED ITERATION MARK

These characters, if replaced, are secondary equal to
the replacing kana, while ternary not equal to.

=over 4

=item KATAKANA-HIRAGANA PROLONGED SOUND MARKs

The PROLONGED MARKs (including the halfwidth equivalent)
are repleced to a normal vowel or nasal
katakana corresponding to the preceding kana if exists.

  e.g.,

  'カー'   to 'カア'
  'びー'   to 'びイ'
  'きゃー' to 'きゃア'
  'ピュー' to 'ピュウ'
  'ンー'   to 'ンン'
  'んー'   to 'んン'

=item HIRAGANA- and KATAKANA ITERATION MARKs

The ITERATION MARKs (VOICELESS) are repleced
to a normal kana corresponding to the preceding kana if exists.

  e.g.,

  'かゝ'   to 'かか'
  'ドゝ'   to 'ドと'
  'んゝ'   to 'んん'
  'カヽ'   to 'カカ'
  'ばヽ'   to 'ばハ'
  'プヽ'   to 'プフ'
  'ヴィヽ' to 'ヴィイ'
  'ピュゝ' to 'ピュゆ'

=item HIRAGANA- and KATAKANA VOICED ITERATION MARKs

The VOICED ITERATION MARKs are repleced to a voiced kana
corresponding to the preceding kana if exists.

  e.g.,

  'はゞ' to 'はば'
  'プゞ' to 'プぶ'
  'プヾ' to 'プブ'
  'こヾ' to 'こゴ'
  'ウヾ' to 'ウヴ'

=item Cases of no replacement

Otherwise, no replacement occurs. Especially in the
cases when these marks follow any character except kana.

The unreplaced characters are primary greater than any kana.

  e.g.  CJK Ideograph followed by PROLONGED SOUND MARK
        Digit followed by ITERATION MARK
        'アヾ' ('ア' has no voiced variant)

=item Example

For example, the Japanese string C<'パール'> (C<Perl> in kana)
has three collation elements: C<KATAKANA PA>,
C<PROLONGED SOUND MARK replaced by KATAKANA A>, and C<KATAKANA RU>.

  e.g.,

   'パール' is converted to 'パアル' by replacement;
     primary equal to 'はある';
     condary equal to 'ぱある', greater than 'はある';
     tertiary equal to 'ぱーる', lesser than 'パアル';
     and quartenary greater than 'ぱーる'.

=back

=head2 Conformance to the Standard [JIS X 4061, 6.2]

  (1) charset: Shift-JIS.

  (2) No limit of the number of characters in the string considered
      to collate.

  (3) No character class is added.

  (4) The following characters are added as collation elements.

      IDEOGRAPHIC SPACE is added in the space class.

      ACUTE ACCENT, GRAVE ACCENT, DIAERESIS, CIRCUMFLEX ACCENT
      are added in 'kijutsu kigou'.

      APOSTROPHE, QUOTATION MARK are added in 'kakko kigou'.

      HYPHEN-MINUS is added in 'gakujutsu kigou'.

  (5) Collation of Latin alphabets with macron and with circumflex
      is not supported.

  (6) Kanji Classes:
       i) the minimal kanji class (Five kanji-like chars).
       ii) the basic kanji class (Levels 1 and 2 kanji of JIS)..

=head1 AUTHOR

SADAHIRO Tomoyuki <SADAHIRO@cpan.org>

  http://homepage1.nifty.com/nomenclator/perl/

Copyright (C) 2001-2003. All rights reserved.

This module is free software; you can redistribute it
and/or modify it under the same terms as Perl itself.

=head1 SEE ALSO

=over 4

=item JIS X 4061

Collation of Japanese character strings

=item JIS X 0201

7-bit and 8-bit coded character sets for information interchange

=item JIS X 0208

7-bit and 8-bit double byte coded KANJI sets for information interchange

=item JIS X 0221

Information technology - Universal Multiple-Octet Coded
Character Set (UCS) - part 1 : Architectute and Basic Multilingual Plane

=item Japanese Industrial Standards Committee (JISC)

L<http://www.jisc.go.jp/>

=item Japanese Standards Association (JSA)

L<http://www.jsa.or.jp/>

=item L<ShiftJIS::String>

=item L<ShiftJIS::Regexp>

=back

=cut
