# this is Kwid
This document describes a processing implementation strategy for parsing
Perldoc dialects into events that can become a uniform DOM.

NOTE: The name 'Perldoc' is chosen to describe any dialect of the
      formatting model formerly know as Pod (including Pod itself).
      This disambiguates Pod the syntax from Pod the overall
      documentation model.

The architecture is layered as such:

   source text  -> 
     (scanner)  -> 
   text stream  ->
     (parser)   ->
   parse events ->
     (receiver) ->
   representation

where parenthesized elements are /processors/ and unparenthesized
elements are /states/. Each is described in further detail:

.list
- source text

The source text can come in various forms:

* An open filehandle
* A named Perl file
* A named Kwid file
* A named Pod file
* A text string

The text does not typically come from more than one source, but if
there is a standard include mechanism it is not handled at this point;
but easy mechanisms for re-

- scanner

The scanner is responsible for taking a source text and manipulating it
into a uniform text stream that the parser can read from. The scanner
has the following API:

-- new(file_name|file_handle|string, [type_hint])  Create a new scanner object
-- type()  Return the top level stream dialect
-- read(N)  Returns N block(s) of text or undef on EOF
-- eos()  True if the current section / block / whatever is finished
-- read(separator, [chomp])  Returns a block of text terminated by the
   separator or undef on EOF. Two common cases of separator are "\n" for
   line mode and "" for paragraph mode. The separator is chomped if the
   `chomp` argument is true.


- text stream

The normalized text for parsing by a given dialect's parser.

Normalizing means removing the parts of the original stream that aren't part
of the Perldoc and wrapping blocks that are of a differing dialect than the
/type/ of the stream into the top level doc's block escaping syntax.

Here is a example using a Perl program containing both Pod and Kwid dialects.
Given the following perl file `question.pl`:

    #!/usr/bin/perl

    =head1 DESCRIPTION

    Just another C<Perl> script!

    =cut
    
    my $question = find_question("42");

    =doc.kwid == `find_question(String)`

    Given an /answer/, return the *question*!

    =cut

    printf "The Question is '%s'\n", $question;

    =head2

    That's it...

If you create a scanner object with:

    my $scanner = Perldoc::Scanner->new('question.pl');

The scanner would produce the following stream:

    =head1 DESCRIPTION

    Just another C<Perl> script!

    =begin kwid
    
    my $question = find_question("42");

    =doc.kwid == `find_question(String)`

    Given an /answer/, return the *question*!

    =end kwid

    printf "The Question is '%s'\n", $question;
   
    =head2

    That's it...

and the scanner's `type` attribute would be set to 'Pod' since that is the
type of the first section encountered in the Perl code.

- parser

A parser reads a text stream and creates event based on the content of that
stream. Events should be detected with minimum lookahead. 

When a given parser encounters text that must be handled by another type of
parser (either a dialect or a special opaque handler) it passes the scanner
and receiver context to the new parser object and that parser takes over. It
must also pass along 'containing rules' so that the new parser knows when to
stop.

If a parser cannot locate a parsing class that can handle the sub-parse, it
should read all the text up until the ending rule for that section, and create
an opaque_object event with the original text fully intact. That way a
downstream processor may be able to parse the text further.

The parser has the following api:

-- new(receiver, scanner, [container-ending-rules])  New parser object
-- parse()  Begin parsing

- parse events 

These are methods in the receiver API:

-- start_document()
-- start_element(type, {})
-- characters(data)
-- end_element([type])
-- processing_instruction(pi, {})
-- ignorable_whitespace(data)
-- end_document()
# I need major help finishing this list

- receiver

A receiver is a process that receives events and does something with them.
Here are some common somthings:

* Produce a DOM
* Produce HTML
* Produce Prettied Text
* Produce LaTeX
* Produce a compact "bytecode"
* Produce POD
* Produce Kwid
* Act as a filter: pass events onto another receiver, possibly mutating
  the original events.

- representation

This is just a term for some form other than the original. It may be
serialized or an in memory graph.
..list
