=head1 NAME

YATT::Lite::XHF::Syntax - Extended Header Fields (XHF) format.

=for code perl

=head1 SYNOPSIS

  require YATT::Lite::XHF;

  my $parser = YATT::Lite::XHF->new(string => <<'END');
  # Taken from http://docs.ansible.com/YAMLSyntax.html#yaml-basics
  name: Example Developer
  job: Developer
  skill: Elite
  employed: 1
  foods[
  - Apple
  - Orange
  - Strawberry
  - Mango
  ]
  languages{
  ruby: Elite
  python: Elite
  dotnet: Lame
  }

  name: hkoba
  languages{
  yatt: Elite?
  }
  END
  
  # read() returns one set of parsed result by one paragraph, separated by \n\n+.
  # In array context, you will get a flattened list of items in one paragraph.
  # (It may usually be a list of key-value pairs, but you can write other types)
  # In scalar context, you will get a hash struct.
  while (my %hash = $parser->read) {
    print Dumper(\%hash), "\n";
  }

=head1 DESCRIPTION

Extended Header Fields (B<XHF>) format, which I'm defining here,
is a data format based on L<Email header|http://tools.ietf.org/html/rfc2822#section-2.2>
(and L<HTTP header|http://tools.ietf.org/html/rfc2616#section-4.2>)
, with extension to carry I<nested data structures>.
To load XHF files/strings, use L<YATT::Lite::XHF>.

Note: Although there is a serializer for XHF (L<YATT::Lite::XHF::Dumper>),
XHF is specifically designed to help programmers B<writing test data>
for unit tests, instead of to be a perfect serializer for perl
(ie. XHF doesn't support self-referencing data structures.
It is not my design goal).
If you want such complex serializer, you should use L<YAML> family, L<Storable> and such instead.

=head2 Minimum escaping for value of name-value pairs
X<minimum-escaping>

For simplest cases, YAML and XHF may look fairly similar. For example,
a hash structure C<< {foo => 1, bar => 2} >> can be written in a same way
both in YAML and in XHF:

  foo: 1
  bar: 2

However, if you serialize a structure C<< {x => [1, 2, "3, 4"], y => 5} >>,
you will notice significant differences.

B<In XHF>, above will be written as:

  {
  x[
  - 1
  - 2
  - 3, 4
  ]
  y: 5
  }


In contrast B<in YAML>, same structure will be written as:

  ---
  x:
    - 1
    - 2
    - '3, 4'
  y: 5

The differences are:

=over 4

=item * XHF uses B<parens> C< {} [] >. YAML uses B<indents>.

=item * XHF can represent C<3, 4> as is. YAML B<needs to escape> it like C<'3, 4'>.

=back

=head3 Multi-line text value and verbatim text
X<multi-line-text>

In XHF, you I<only> need to escape C<\n>
(and leading/trailing C<SPACE>, C<TAB>, if you need)
for each value-part.
In other words,
there is I<no syntax> for value-part so you don't need to worry
about which characters must be escaped.

=over 4

=item How to escape newlines in the middle.
X<newline-escaping> X<line-continuation>

Just substitute all C<"\n"> with C<"\n "> like C<< s/\n/\n /g >>.

eg. C<< { foo => "1\n2\n\n3", bar => 4 } >> can be written as:

   foo: 1
    2
    
    3
   bar: 4

=item How to escape leading/trailing spaces/tabs/newlines.
X<verbatim-text>

Just start value with C<":\n"> and follow same escaping rule for C<"\n">.

eg. C<< { foo => "  x  ", bar => "\n\ny\n\n" } >> can be written as:

  foo:
    x  
  bar:
   
   
   y
   
   


=back

=head2 Name-value pair can be written as two items separately
X<pair-by-two-items>

In contrast to value-part, name-part has syntax restriction.
name-part of XHF can contain only C<[[:alnum:]]>, C<"-">, C<".">, C<"/">
and some additional chars(see I<field-name> definition
in L</BNF>).
However, you can use two C<- > items to write name-value pairs interchangeably.
So again, whenever you are not sure about allowed char, you can use C<- > notation and only escape C<\n>.

   # For example, following block:

   foo: 1
   bar: 2

   # can be written as following:

   - foo
   - 1
   - bar
   - 2

eg. C<< { "foo bar" => "baz" } >> can be written as:

  {
  - foo bar
  - baz
  }

And C<< { "\n  foo\nbar  \n" => "baz" } >> can be written as:

  {
  -
  
     foo
   bar
  
  - baz
  }


For nested elements, same applies.

   foo{
   x: 1
   y: 2
   }
   baz[
   - z
   ]

   # can be written instead as following:

   - foo
   {
   x: 1
   y: 2
   }
   - baz
   [
   - z
   ]

   # or even like following:

   - foo
   {
   - x
   - 1
   - y
   - 2
   }
   - baz
   [
   - z
   ]

Also, you can put C<key: value> notation in arrays, like following:

  [
  foo: 1
  bar: 2
  ]

  # above is equal to following

  [
  - foo
  - 1
  - bar
  - 2
  ]

=head2 Container Agnostic List
X<container-agnostic>

Another important difference (you might notice in previous examples)
is at B<container type selection> (array or dict).
In XHF, name-value separator determines "type of I<value>"
instead of "type of I<surrounding container>".

In XHF, following block

  foo: 1
  bar: 2

just represents C<< ( foo => 1, bar => 2 ) >>,
which is flattened list of 4 items. This itself do not determine
surrounding container type. Then you can choose outermost container type like

  my %dict = $parser->read;

or

  my @array = $parser->read;

When you call C<read()> via scalar context, you will get a dictionary
(or an error when the block has odd number of items).

  my $dict = $parser->read;

In contrast in YAML, C<:> always means I<map(dictionary)>. So, above
will be always C<< +{ foo => 1, bar => 2 } >>.

=head3 Ordered kv-pair list with key duplicates (limited)
X<ordered-kv-pairs>

Since outermost xhf-block means flattened list, you can use XHF
to write down ordered key-value pair list with key duplicates,
like following:

  foo: 1
  foo: 2
  foo: 3
  bar: x
  bar: y

If you read above with

  my @array = $parser->read;

you can get C<< @array == (foo => 1, foo => 2, foo => 3, bar => 'x', bar => 'y') >> exactly.

This is important for some kind of test data (eg. HTTP query parameters
and some of Email header fields like "Received"). For example,
above is (equivalent of) valid output from following html form in HTTP:

=for code html

  <input type="checkbox" name="foo" value="1">
  <input type="checkbox" name="foo" value="2">
  <input type="checkbox" name="foo" value="3">
  <input type="checkbox" name="bar" value="x">
  <input type="checkbox" name="bar" value="y">


=for code perl

Note: currently, nested elements are deserialized as ordinally perl hash and array, so this order/dup-key preservation only works for outermost list.

=head2 Paragraph based block stream (with comment skipping)
X<paragraph-stream>

XHF input stream is delimited by consecutive empty-line(s) C<"\n\n+">
(like Email header and HTTP header),
designed to work well with traditional "paragraph mode"
multi-line record format. For more about paragraph mode, see L<perl -00|perlrun/-0> and L<Setting $RS to ""|perlvar/$RS> in perldoc.

Note: in XHF, "comment-only" blocks are skipped silently. For example:

  foo: 1
  bar: 2

  # Hey, here is a comment only block!


  baz: 3
  qux: 4

Then this script:

  my @records;
  push @records, $_ while $_ = $parser->read;

will result C<< @records == ({foo => 1, bar => 2}, {baz => 3, qux => 4}) >>.

=head3 How to put metainfo as optional (comment-only) record
X<howto-skip_comment>

In rare case, you may want to prepend optional meta record in single stream.
If you really want to do this, you can use "comment only" block to represent
empty record and read it with C<< read(skip_comment => 0) >> like following:

  # This is metainfo. To put test => 1, please remove leading "# " below:
  # test: 1


  # This is body1
  foo: 1
  bar: 2

  # This is body2
  foo: 3
  bar: 4



Then

  if (my @meta = $parser->read(skip_comment => 0)) {
    # process metainfo. You may get (test => 1).
  }
  while (my @content = $parser->read) {
    # process body1, body2, ...
  }

=head1 COMPLEX EXAMPLE, compared with YAML
X<complex-example>

Here is a more dense example B<in XHF>:

  name: hkoba
  # (1) You can write a comment line here, starting with '#'.
  job: Programming Language Designer (self-described;-)
  skill: Random
  employed: 0
  foods[
  - Sushi
  #(2) here too. You don't need space after '#'. This will be good for '#!'
  - Tonkatsu
  - Curry and Rice
  [
  - More nested elements
  ]
  ]
  favorites[
  # (3) here also.
  {
  title: Chaika - The Coffin Princess
  # (4) ditto.
  heroine: Chaika Trabant
  }
  {
  title: Witch Craft Works
  heroine: Ayaka Kagari
  # (5) You can use leading "-" for hash key/value too (so that include any chars)
  - Witch, Witch!
  - Tower and Workshop!
  }
  # (6) You can put NULL(undef) like below. (equal space sharp+keyword)
  = #null
  ]

Above will be loaded like following structure:

  $VAR1 = {
          'foods' => [
                     'Sushi',
                     'Tonkatsu',
                     'Curry and Rice',
                     [
                       'More nested element'
                     ]
                   ],
          'job' => 'Programming Language Designer (self-described;-)',
          'name' => 'hkoba',
          'employed' => '0',
          'skill' => 'Random',
          'favorites' => [
                         {
                           'heroine' => 'Chaika Trabant',
                           'title' => 'Chaika - The Coffin Princess'
                         },
                         {
                           'title' => 'Witch Craft Works',
                           'heroine' => 'Ayaka Kagari',
                           'Witch, Witch!' => 'Tower and Workshop!'
                         },
                         undef
                       ]
        };


Above will be written B<in YAML> like below (note: inline comments are omitted):

  ---
  employed: 0
  favorites:
    - heroine: Chaika Trabant
      title: 'Chaika - The Coffin Princess'
    - 'Witch, Witch!': Tower and Workshop!
      heroine: Ayaka Kagari
      title: Witch Craft Works
    - ~
  foods:
    - Sushi
    - Tonkatsu
    - Curry and Rice
    -
      - More nested element
  job: Programming Language Designer (self-described;-)
  name: hkoba
  skill: Random


This YAML example clearly shows how you need to escape strings quite randomly,
e.g. see above value of C<< $VAR1->{favorites}[0]{title} >>.
Also the key of C<< $VAR1->{favorites}[1]{'Witch, Witch!'} >> is nightmare.

I don't want to be bothered by this kind of escaping.
That's why I made XHF.

=head1 FORMAT SPECIFICATION
X<XHF-FORMAT>

XHF are parsed one paragraph by one.
Each paragraph can contain a set of C<xhf-item>s.
Every xhf-items start from a fresh newline, ends with a newline
and is basically formed like one of followings:

  <name> <type-sigil> <sep> <value>         (name-value pair)

  <type-sigil> <sep> <value>                (standalone value)

C<type-sigil> defines type of C<value>.
C<sep> is usually one of logical whitespace chars
where C<space>, C<tab> and C<newline>
(newline is used for L<verbatim text|/verbatim-text>).
But for block items(dict/array), only C<newline> is allowed.

Here is all kind of type-sigils:

=over 4

=item C<"name:"> then C<" "> or C<"\n">

C<":"> is for ordinally text with name. I<MUST> be prefixed by C<name>. C<sep> can be any of WS.

=item C<"-"> then C<" "> or C<"\n">

C<"-"> is for ordinally text without name. I<CANNOT> be prefixed by C<name>.

(Note: Currently, C<","> works same as C<"-">. This feature is arguable.)

=item C<"{"> then C<"\n">

=item C<"name{"> then C<"\n">

C<"{"> is for dictionary block (C< { %HASH } > container). I<Can> be prefixed by C<name>.

I<MUST> be closed by C<"}\n">. Number of elements I<MUST> be even.


=item C<"["> then C<"\n">

=item C<"name["> then C<"\n">

C<"["> is for array block. (C< [ @ARRAY ] > container). I<Can> be prefixed by C<name>.

I<MUST> be closed by C<"]\n">

=item C<"="> then C<" "> or C<"\n">

=item C<"name="> then C<" "> or C<"\n">

C<"="> is for special values. I<Can> be prefixed by C<name>.

Currently only C<#undef> and its synonym C<#null> is defined.

=item C<"#">

C<"#"> is for embedded comment line. I<CANNOT> be prefixed by C<name>.

=back

=head2 XHF Syntax definition in extended BNF
X<syntax> X<BNF>

Here is a syntax definition of XHF in extended BNF
(I<roughly> following L<ABNF|https://tools.ietf.org/html/rfc5234>.)

  xhf-block       = 1*xhf-item

  xhf-item        = field-pair / single-text
                   / dict-block / array-block / special-expr
                   / comment

  field-pair      = field-name  field-value

  field-name      = 1*NAME *field-subscript

  field-subscript = "[" *NAME "]"

  field-value     = ":" text-payload / dict-block / array-block / special-expr

  text-payload    = ( trimmed-text / verbatim-text ) NL

  trimmed-text    = SPTAB *( 1*NON-NL / NL SPTAB )

  verbatim-text   = NL    *( 1*NON-NL / NL SPTAB )

  single-text     = "-" text-payload

  dict-block      = "{" NL *xhf-item "}" NL

  array-block     = "[" NL *xhf-item "]" NL

  special-expr    = "=" SPTAB known-specials NL

  known-specials  = "#" ("null" / "undef")

  comment         = "#" *NON-NL NL

  NL              = [\n]
  NON-NL          = [^\n]
  SPTAB           = [\ \t]
  NAME            = [0-9A-Za-z_.-/~!]

=head2 Some notes on current definition
X<notes-on-current-def>

=over 4

=item field-name, field-subscript

B<field-name> can contain C</>,  C<.>, C<~> and C<!>.
Former two are for file names (path separator and extension separator).
Later two (and B<field-subscript>) are incorporated just to help
writing test input/output data for L<YATT::Lite>,
so these can be arguable for general use.


=item trimmed-text vs verbatim-text

If B<field-name> is separated by C<": ">, its B<field-value> will be trimmed
their leading/trailing spaces/tabs.
This is useful to handle hand-written configuration files.

But for some software-testing purpose(e.g. templating engine!),
this space-trimming makes it impossible to write exact input/output data.

So, when B<field-sep> is NL, field-value is not trimmed.

=item LF vs CRLF

Currently, I'm not so rigid to reject the use of CRLF.
This ambiguity may harm use of XHF as a serialization format, however.

=item C<","> can be used in-place of C<"-">.

This feature also may be arguable for general use.

=item C<":"> without C<name> was valid, but is now deprecated.

Previously valid

  : bar

which represents C<< ( "" => "bar" ) >>, is now invalid.
Please use two C<"- "> items like following:

  - 
  - bar

XXX: Hmm, should I provide deprecation cycle? Are there someone
already used XHF to serialize important data even before having this manual?
If so, please contact me. I will add an option to allow this.

=item line-continuation is valid.

Although L<line-continuation|/line-continuation> is obsoleted
in L<HTTP headers|http://tools.ietf.org/html/rfc7230#section-3.2.4>,
line-continuation will be kept valid in XHF spec. This is my preference.

=back


=head1 AUTHOR

"KOBAYASI, Hiroaki" <hkoba@cpan.org>

=head1 LICENSE

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
