\input texinfo  @c -*-texinfo-*-

@c %**start of header
@setfilename guile-reader.info
@settitle A Reader Framework for Guile
@footnotestyle end
@setchapternewpage odd
@c %**end of header

@set EDITION 1
@include version.texi

@copying
This file documents Guile-Reader.

Copyright @copyright{} 2005, 2006, 2007, 2008, 2009, 2012, 2015, 2017 Ludovic Court@`es

Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions,
except that this permission notice may be stated in a translation
approved by the Free Software Foundation.
@end copying

@dircategory The Algorithmic Language Scheme
@direntry
* Guile Reader: (guile-reader).      A Reader Framework for Guile.
@end direntry


@titlepage
@title A Reader Framework for Guile
@subtitle for Guile-Reader @value{VERSION}
@author Ludovic Court@`es

@page
@vskip 0pt plus 1filll
Edition @value{EDITION} @*
@c Updated for Guile Reader @value{VERSION} @*
@value{UPDATED} @*

@insertcopying
@end titlepage

@contents

@c *********************************************************************
@node Top, Introduction, (dir), (dir)
@top A Reader Framework for Guile

This document describes Guile-Reader version @value{VERSION}, for
GNU@tie{}Guile's 3.0 stable series, as well as the legacy 1.8, 2.0 and
2.2 series.  It was last updated in @value{UPDATED-MONTH}.

This documentation should be mostly complete.  Details about the C API
are omitted from this manual.  They can be found in the public header
files.

@menu
* Introduction::                The goals of Guile-Reader.
* Overview::                    How it works.
* Quick Start::                 For the impatient.
* API Reference::               Using Guile-Reader.
* Internals::                   Performance considerations.

* Concept Index::               Concepts.
* Function Index::              Scheme procedures and C functions.
@end menu


@c *********************************************************************

@node Introduction, Overview, Top, Top
@chapter Introduction

Guile currently provides limited extensibility of its reader, by means
of @code{read-hash-extend} (@pxref{Reader Extensions,
@code{read-hash-extend},, guile, Guile Reference Manual}), for
instance, or @code{read-enable} (@pxref{Scheme Read,
@code{read-enable},, guile, Guile Reference Manual}).
@uref{http://srfi.schemers.org/srfi-10/srfi-10.html, SRFI-10} tries to
propose a generic, portable, extension mechanism similar to
@code{read-hash-extend} but limited to @code{#,} sequences.  Moreover,
while this may not always be desirable, all these extension facilities
have a global effect, changing the behavior of the sole reader
implementation available at run-time.  This makes it impossible to
have, for instance, one module consider names starting with @code{:}
as symbols, while another considers them as keywords.

Extensions such as the read syntax for SRFI-4 numeric vectors
(@pxref{Uniform Numeric Vectors, SRFI-4 vectors,, guile, Guile
Reference Manual}) had to be added to Guile's built-in C reader.
Syntactic extensions that did not appeal the majority of users, like
Emacs-Lisp vectors, are @code{#ifdef}'d within the reader code and are
not available by default.  Moreover, some extensions are incompatible
with each other, such as the DSSSL keyword syntax and SCSH block
comments (@pxref{Block Comments, SCSH block comments,, guile, Guile
Reference Manual}).  In short the current reader syntax is hardly
extensible.

The idea of Guile-Reader is to provide a framework allowing users to
quickly define readers for whatever syntax (or rather: variant of the
Scheme syntax) they like.  Programs can then provide their own readers
and, thanks to Guile's @code{current-reader} mechanism, have their
code read with this reader.  When using Guile 2.0, readers produced by
Guile-Reader are Unicode-capable; they can read from any
ASCII-compatible encoding, such as UTF-8 or ISO-8859-1.

@cindex SILex
@cindex lexer
While it is much simpler than a full-blown lexer generator such as
@uref{http://www.gnu.org/software/flex/, Flex},
@uref{http://www.iro.umontreal.ca/~dube/, Danny Dub@'e's SILex} and
@uref{http://www.inria.fr/mimosa/fp/Bigloo, Bigloo's RGC}, its simple
programming interface should make it very straightforward to implement
readers, especially for Scheme-like syntaxes.  Best of all, Guile
Reader comes with a library of components that can typically be used
to construct a reader for the Scheme syntax.  And each one of this
components may be reused at will when creating other readers.  On the
other hand, one should be aware that this simpler API comes at the
cost of a lack of consistency in some cases, as outlined later in this
manual (@pxref{Limitations}).

@cindex read table
Common Lisp has a similar mechanism to extend its reader which is
called the @dfn{read
table}.  @uref{http://www.iro.umontreal.ca/~gambit/, Gambit Scheme},
for instance, also provides an implementation of read tables.
However, it appears to have limitations similar to Guile's
@code{read-enable} and @code{read-hash-extend} in terms of
possibilities for syntax extension.  On the other hand, it allows the
reader and writer to be kept consistent, which Guile-Reader does not
address.


@c chapter ----------------------------------------------------------
@node Overview, Quick Start, Introduction, Top
@chapter Overview

@cindex reader library
Guile-Reader allows for the construction of readers capable of
understanding various syntactic variants.  The simplest way to use it
is through its @dfn{reader library} that allows one to pick and choose
various commonly used syntactic extensions to the standard Scheme
syntax (@pxref{Reader Library}).  However, Guile-Reader also provides
a finer-grain programming interface allowing the construction of
virtually any reader, with its own syntactic specificities.  The
following sections focus primarily on this capability.

@cindex reader
@cindex token reader
Before going into the details of the reader framework API, let us have
a quick overview of what this is.  Basically, Guile-Reader introduces
two objects: @dfn{readers} and @dfn{token readers}.  Readers can be
thought of, simply, as procedures like Scheme's @code{read}
(@pxref{Input, R5RS,, r5rs, Revised^5 Report on the Algorithmic
Language Scheme}), i.e., procedures that take one (optional) argument,
namely the port to read from.  We will see later that readers as
defined by Guile-Reader can actually receive two more arguments
(@pxref{Defining a New Reader}).  A reader, like @code{read}, reads a
sequence of characters (the @dfn{external representation} of some
object) and returns a Scheme object.

Token readers (TRs, for short) are the building block of a reader.  A
token reader is basically an association between a character or set of
characters and a procedure to read and interpret a sequence of
characters starting with one of the former.  For instance, in a
standard Scheme reader, the character @code{(} may be associated to a
procedure that reads an S-expression.  Likewise, lower-case and
upper-case letters associated with the appropriate procedure form a
token reader for symbols.

In Guile-Reader, TRs may be written either in Scheme or in C, and they
can even be a reader produced by Guile-Reader itself.  Unless it is a
reader, the procedure (or C function) used to create a TR will receive
four arguments:

@itemize
@item the character that was read and which triggered its call;  in
the S-exp example, this would be @code{(};
@item the port to read from;
@item the reader which performed this invocation;
@item the top-level reader which yielded this invocation.
@end itemize

The next section shows how to get started with Guile-Reader, using a
high-level API.  Details about of the programming interface are given in
the API reference (@pxref{API Reference}).


@c chapter ----------------------------------------------------------
@node Quick Start, API Reference, Overview, Top
@chapter Quick Start

The simplest way to get started and to produce customized readers is
through the high-level API provided by the @code{(system reader
library)} module.  As the name suggests, this module provides a library
of readily usable readers.  The @code{make-alternate-guile-reader}
procedure gives access to these readers.  It can be passed a list of
symbols describing reader options, such as whether you want support for
DSSSL keywords, @uref{http://srfi.schemers.org/srfi-62/srfi-62.html,
SRFI-62 comments}, @uref{http://srfi.schemers.org/srfi-30/srfi-30.html,
SRFI-30 block comments}, etc.

The following example binds to @var{my-reader} a Scheme reader that
supports DSSSL-style keywords and SRFI-62 comments, and that is case
insensitive:

@findex make-alternate-guile-reader

@lisp
(use-modules (system reader library))

(define my-reader
  (make-alternate-guile-reader '(dsssl-keywords
                                 srfi62-sexp-comments
                                 case-insensitive)))
@end lisp

This reader can then be used like the regular @code{read} procedure:

@lisp
(procedure? my-reader)
=> #t

(with-input-from-string "some-symbol" my-reader)
=> some-symbol

(my-reader (open-input-string "MiXeD-CaSe"))
=> mixed-case

(my-reader (open-input-string "(an sexp with a #;srfi-62 comment)"))
=> (an sexp with a comment)

(my-reader (open-input-string "#!some-DSSSL-keyword"))
=> #:some-dsssl-keyword
@end lisp

Most of the time, you will want to use it as the current reader, at
least in the module where you created it.  Fortunately, Guile
provides a mechanism for this, the @code{current-reader} fluid
(@pxref{Loading, @code{current-reader},, guile, Guile Reference
Manual}).  Changing the value of this fluid from a file that is being
loaded will affect the reader use to load it.  In general, you will want
to modify @code{current-reader} not only at run time, but also at
compile time and when the code is evaluated, which can be achieved
using @code{eval-when} (@pxref{Eval When, @code{eval-when},, guile, GNU
Guile Reference Manual}).  For instance:

@vindex current-reader

@lisp
;;; This is the beginning of my Scheme file.  At this point, Guile's
;;; default reader is used (or, alternatively, the one that was
;;; passed as a second parameter to `load').  So, for instance, DSSSL
;;; keywords are _not_ recognized.

;; Let's create our own customized reader...
(use-modules (system reader library))

;; 'eval-when' here is needed to make 'my-reader' accessible at
;; compile time, and to have the 'current-reader' change take
;; effect at compile time.
(eval-when (compile load eval)
  (define my-reader
    (make-alternate-guile-reader '(dsssl-keywords
                                   srfi62-sexp-comments
                                   case-insensitive)))

  ;; Let's make it the current reader.
  (fluid-set! current-reader my-reader))

;; From now on, MY-READER is used to read the rest of this file.  Thus
;; we can happily use the syntactic extensions it implements: DSSSL
;; keywords, SRFI-62 comments and case-insensitivity.

(if (not (keyword? #!dsssl-keyword))
    (error "Something went wrong, this should not happen!"))
@end lisp

The nice thing is that @code{current-reader} is reset to its initial
value when the dynamic extent of @code{load} is left.  In other words,
the loader of the file above is @emph{not} affected by the
@code{fluid-set!} statement.  Reader changes are @emph{hygienic} and
modules can use their own without risking to harm each other.

The full list of options supported by @code{make-alternate-guile-reader}
is shown in @ref{Reader Library}.  However, this option set is quite
limited and you may find yourself wanting a syntactic extension not
available here.  In that case, you will want to build a new reader,
possibly reusing existing reader components known as @dfn{token
readers}, as described in @ref{Defining a New Reader}.


@c chapter ----------------------------------------------------------
@node API Reference, Internals, Quick Start, Top
@chapter API Reference

All the Scheme procedures described below are exported by the
@code{(system reader)} module.  In order to be able to use them, you
will need to import this module first:

@example
(use-modules (system reader))
@end example

A C variant is also available for most of them by including the
declarations available in the @code{<guile-reader/reader.h>} header
file.

@menu
* Token Readers::               The basic components of a reader.
* Readers::                     How to build readers.
@end menu

@node Token Readers, Readers, API Reference, API Reference
@section Token Readers

Basically, token readers are the association of a character or set of
characters and a function that is able to interpret character
sequences that start by one of these characters.  We will see below
how to define new token readers first, and then how to re-use existing
ones.

@menu
* Defining a New Token Reader::  
* TR Calling Convention::       
* Invoking a Reader from a TR::  
* Token Reader Library::        
* Limitations::                 
@end menu

@node Defining a New Token Reader, TR Calling Convention, Token Readers, Token Readers
@subsection Defining a New Token Reader

@cindex token reader
@cindex character specification
@findex make-token-reader

A new token reader object can be created by calling the
@code{make-token-reader} procedure with a @dfn{character
specification} and a procedure.  A character specification defines the
set of characters which should trigger an invocation of the
corresponding procedure.  The character specification may be either:

@itemize
@item a single character;
@item a pair of characters, which is interpreted as a character range;
@item a list of characters, which is interpreted as a set of
characters.
@end itemize

The procedure passed to @code{make-token-reader} may actually be
either a C function or Scheme procedure that takes four arguments
(@pxref{TR Calling Convention}), any ``object'' returned by
@code{token-reader-procedure}, or a reader.  This last option turns
out to be quite helpful.  For example, this is very convenient
when implementing the various Scheme read syntaxes prefixed by the
@code{#} character: one can create a reader for @code{#}, and then
turn it into a token reader that is part of the top-level reader.

The reference for @code{make-token-reader} is given below:

@c Include part of API documentation snarffed from `reader.c'.
@include make-token-reader-doc.texi

The next section explains the token reader calling convention, i.e.,
how the @var{proc} argument to @code{make-token-reader} is invoked.


@node TR Calling Convention, Invoking a Reader from a TR, Defining a New Token Reader, Token Readers
@subsection Token Reader Calling Convention

@cindex calling convention
A token reader's procedure is passed four arguments:

@itemize
@item the character that was read and which triggered its call;  in
the S-exp example, this would be @code{(};
@item the port to read from;
@item the reader which performed this invocation, i.e., either an
@code{scm_reader_t} object (if the token reader is written in C) or a
four-argument Scheme procedure (if the token reader is written in
Scheme);
@cindex top-level reader
@item the top-level reader which yielded this invocation and which may
be different from the previous argument in the case a token reader was
made from a reader; the use of these two arguments will be detailed in
the next section, @ref{Invoking a Reader from a TR}.
@end itemize

It must return a Scheme object resulting from the interpretation of
the characters read.  It may as well raise an error if the input
sequence is corrupt.  Finally, it may return @code{*unspecified*}, in
which case the calling reader will not return and instead continue
reading.  This is particularly useful to define comment token readers:
a TR that has just read a comment will obviously not have any sensible
Scheme object to return, and a reader is not expected to return
anything but a ``real'' Scheme object.  A token reader for Scheme's
@code{;} line comments may be defined as follows:

@smallexample
(make-token-reader #\; read-a-line-and-return-unspecified)
@end smallexample

This behavior may, however, be overridden by passing
@code{make-token-reader} a third argument (called @var{escape?}):

@smallexample
(make-token-reader #\; read-a-line-and-return-unspecified #t)
@end smallexample

@cindex SCSH block comments
A reader that includes this TR will return @code{*unspecified*} once a
line comment has been read.  This is particularly useful, for
instance, when implementing @code{#!} block comments (@pxref{Block
Comments, SCSH block comments,, guile, Guile Reference Manual}, for
more information) as a TR attached to @code{#\!} within the @code{#\#}
sub-reader (@pxref{Defining a New Token Reader}).

Finally, the procedure passed to @code{make-token-reader} may be
@code{#f}, in which case the resulting TR will just have the effect of
ignoring the characters it is associated to.  For instance,
handling white spaces may be done by defining a TR like this:

@smallexample
(make-token-reader '(#\space #\newline #\tab) #f)
@end smallexample


@node Invoking a Reader from a TR, Token Reader Library, TR Calling Convention, Token Readers
@subsection Invoking a Reader from a Token Reader

As seen in section @xref{Defining a New Token Reader}, token readers
are systematically passed to readers when invoked.  The reason why
this may be useful may not be obvious at first sight.

Consider an S-exp token reader.  The TR itself doesn't have sufficient
knowledge to read the objects that comprise an S-exp.  So it needs to
be able to call the reader that is being used to actually read those
objects.

The need for the @var{top-level-reader} argument passed to token
readers may be illustrated looking at the implementation of the vector
read syntax (@pxref{Vector Syntax,,, guile, Guile Reference Manual}).
One may implement the vector reader as a token reader of the @code{#}
sub-reader (@pxref{Defining a New Token Reader}).  The vector token
reader may be implemented like this:

@example
(lambda (chr port reader top-level-reader)
  ;; At this point, `#' as already been read and CHR is `(',
  ;; so we can directly call the regular S-expression reader
  ;; and convert its result into a vector.
  (let ((sexp-read (token-reader-procedure
                    (standard-token-reader 'sexp))))
    (apply vector
           (sexp-read chr port reader))))
@end example

When this procedure is invoked, @var{reader} points to the @code{#}
sub-reader.
@c FIXME: Currently the token_reader_proc_apply supports only
@c 3 arguments, see the code for details.  The explanation below is
@c for the day when it supports the fourth argument.
@c
@c Clearly, in order to read the symbols that comprise the
@c list, @var{sexp-read} should @emph{not} invoke @var{reader} because
@c @var{reader} only knows about @code{#}-prefixed object syntaxes.  For
@c this reason, in order to be consistent in re-usable, the S-exp reader
@c must call @var{top-level-reader} which points to the top-level reader,
@c i.e., the reader which yielded the invocation of the @code{#}
@c sub-reader.

@node Token Reader Library, Limitations, Invoking a Reader from a TR, Token Readers
@subsection Token Reader Library

@cindex token reader library
Guile-Reader comes with a number of re-usable token readers.
Together, they might be assembled to form a complete Scheme reader
equivalent to that of Guile (@pxref{Reader Library}).  Or they can be
used individually in any reader.

The @code{standard-token-reader} procedure takes a symbol that names a
standard TR from the library and returns it (or @code{#f} if not
found).  Currently, the available TRs are:

@c Include the documentation that was generated using `extract-doc.scm'.
@include token-reader-doc.texi

As can be inferred from the above two lists, reading character
sequences starting with the @code{#} characters can easily be done by
defining a sub-reader for that character.  That reader can then be
passed to @code{make-token-reader} as the procedure attached to
@code{#}:

@example
(define sharp-reader
  (make-reader (map standard-token-reader
                    '(boolean character
                      number+radix keyword
                      srfi-4
                      block-comment))))

(define top-level-reader
  (make-reader (list (make-token-reader #\# sharp-reader)
                     ...
                     )))
@end example

The procedures available to manipulate token readers are listed below:

@c Include part of API documentation snarffed from `reader.c'.
@include token-reader-api-doc.texi

@node Limitations,  , Token Reader Library, Token Readers
@subsection Limitations

This section describes the main limitations and common pitfalls
encountered when using Guile-Reader.

@menu
* Token Delimiters::            Dealing with token delimiters.
* Overlapping Token Readers::   Token readers that syntactically overlap.
@end menu

@node Token Delimiters, Overlapping Token Readers, Limitations, Limitations
@subsubsection Token Delimiters

@cindex token delimiters
As can be seen from the previous section, there exist, for instance,
an surprisingly high number of symbol token readers.  The reason for
this is that different syntax variants define different @dfn{token
delimiters}.  Token delimiters are characters that help the reader
determine where tokens that require implicit termination do terminate.
Quoting R5RS (@pxref{Lexical structure, R5RS' lexical structure,,
r5rs, Revised^5 Report on the Algorithmic Language Scheme}):

@quotation
Tokens which require implicit termination (identifiers, numbers,
characters, and dot) may be terminated by any <delimiter>, but not
necessarily by anything else.
@end quotation

@cindex R5RS
@cindex R6RS
R5RS defines token delimiters as one of the following: a whitespace, a
parentheses, a quotation mark (@code{"}) or a semi-colon (@code{;})
character.  On the other hand, R6RS, which is to support the ability
to use square brackets instead of parentheses for S-expressions, also
considers square brackets as token delimiters.  Likewise, if we were
to support curly braces to enclose S-expressions, then curly braces
would need to be considered as token delimiters too.

For this reason, the token reader library comes with several symbol
token readers: the @code{guile-symbol-} family does not consider
square brackets as delimiters while the @code{r6rs-symbol-} family
does, the @code{brace-free-} TR family considers curly braces as
delimiters but not square brackets, etc.  Similarly, several variants
of number TRs are available.  This is due to the fact that number TRs
may return symbols in corner cases like symbol names starting with a
number.

However, although keywords must also comply with the token delimiters
rules, there is only one keyword TR (called @code{keyword}).  The
reason for this is that this TR relies on the top-level reader's
symbol reader to read the symbol that makes up the keyword being read.

@cindex pitfall
In the current design of Guile-Reader, this token delimiter issue
creates a number of pitfalls when one is willing to change the current
delimiters.  In particular, one has to be very careful about using TRs
that consistently assume the same token delimiters.

A ``real'' lexer generator such as Danny Dub@'e's SILex avoids such
issues because it allows the definition of tokens using regular
expressions.  However, its usage may be less trivial than that of
Guile-Reader.

@node Overlapping Token Readers,  , Token Delimiters, Limitations
@subsubsection Overlapping Token Readers

As can be seen from the descriptions of the standard token readers
(@pxref{Token Reader Library}), token readers sometimes ``overlap'',
i.e., the set of input strings they match overlap.  For instance, the
@code{boolean} token reader should match @code{#t}, @code{#T},
@code{#f} or @code{#F}.  However, the @code{srfi-4} token reader also
needs to match floating-point numeric vectors such as @code{#f32(1.0
2.0 3.0)}.  Similarly, strings like @code{1} are, logically, handled
by the @code{guile-number} (or similar) token reader; however, since a
string like @code{1+} should be recognized as a @emph{symbol}, rather
than a number, it must then be passed to one of the symbol token
readers.

In those two cases, the input sets of those two token readers
@emph{overlap}.  In order for the resulting reader to work as
expected, the two overlapping token readers need to somehow
@emph{cooperate}.  In the first example, this is achieved by having
the @code{srfi-4} TR read in strings starting with @code{#f} or
@code{#F} and passing them to the @code{boolean-srfi-4} TR if need be.
In the second case, this is done by having number TRs (e.g.,
@code{guile-number}) explicitly check for non-digit characters and
return a symbol instead of a number when a non-digit is encountered.

It should be obvious from these two examples that this limitation
impedes full separation of the various TRs.  Fortunately, there are
not so many cases where such overlapping occurs when implementing
readers for R5RS-like syntaxes.  The implementation of
@code{make-alternate-guile-reader} (@pxref{Reader Library}) shows how
such problems have been worked around.

@cindex lexer
Lexer generators such as @uref{http://www.gnu.org/software/flex/,
Flex}, @uref{http://www.iro.umontreal.ca/~dube/, SILex} and Bigloo's
RGC (@pxref{Regular Parsing, Bigloo's RGC,, bigloo, Bigloo@comma{} A
``Practical Scheme Compiler''---User Manual}) obviously do not have
this problem: all possible ``token'' types are defined using regular
expressions and the string-handling code (e.g., code that converts a
string into a Scheme number) is only invoked once a full matching
string has been found.


@node Readers,  , Token Readers, API Reference
@section Readers

Guile-Reader is about defining readers.  Continuing to read this
manual was definitely a good idea since we have finally reached the
point where we will start talking about how to define new readers.

@menu
* Defining a New Reader::       
* Reader Library::              
* Compatibility and Confinement::  
@end menu

@node Defining a New Reader, Reader Library, Readers, Readers
@subsection Defining a New Reader

@cindex reader
Roughly, a reader is no more than a loop which reads characters from a
given port, and dispatches further interpretation to more specific
procedures.  Written in Scheme, it could resemble something like:

@lisp
(define (my-reader port)
  (let loop ((result *unspecified*))
    (let ((the-char (getc port)))
      (case the-char
        ((#\()  (my-sexp-token-reader the-char port my-reader)))
        ((#\0 #\1 #\2 #\3 #\4 #\5 #\6 #\7 #\8 #\9)
         (my-number-token-reader the-char port my-reader))
        (else
         (error "unexpected character" the-char))))))
@end lisp

@findex make-reader
Using Guile-Reader, this is done simply by providing a list of token
readers to the @code{make-reader} procedure, as in the following
example:

@lisp
(define my-reader
  (make-reader (list (make-token-reader #\( my-sexp-token-reader)
                     (make-token-reader '(#\0 . #\9)
                                        my-number-token-reader))))
@end lisp

However, the procedure returned by @code{make-reader} is different
from the hand-written one above in that in takes two additional
optional arguments which makes it look like this:

@lisp
(define (my-reader port faults-caller-handled? top-level-reader)
  (let loop ((the-char (getc port)))
    (case the-char
      ...
      (else
        (if (not faults-caller-handled?)
            (error "unexpected character" the-char)
            (ungetc the-char) ;; and return *unspecified*
            )))))
@end lisp

Therefore, by default, @code{my-reader} will raise an error as soon as
it reads a character that it does not know how to handle.  However, if
the caller passes @code{#t} as its @code{faults-caller-handled?}
argument, then @code{my-reader} is expected to ``unget'' the faulty
character and return @code{*unspecified*}, thus allowing the caller to
handle the situation.

This is useful, for instance, in the S-exp token reader example: the
S-exp token reader needs to call its calling reader in order to read
the components between the opening and closing brackets;  however, the
calling reader may be unable to handle the @code{#\)} character so the
S-exp token reader has to handle it by itself and needs to tell it to
the reader.

@c Include part of API documentation snarffed from `reader.c'.
@include make-reader-doc.texi

Currently, the flags that may be passed to @code{make-reader} are the
following:

@itemize

@item @code{reader/record-positions} will yield a reader that records
the position of the expression read, which is mostly useful for
debugging purposes; this information may then be accessed via source
properties (@pxref{Procedure Properties, source properties,, guile,
Guile Reference Manual}).

@item @code{reader/lower-case} will have the yielded reader convert to
lower-case all the letters that it reads; note that this is not
sufficient to implement symbol case-insensitivity as shown in @ref{Reader
options,,, guile, Guile Reference Manual}.  For this, the token
reader(s) that read symbols must also convert all subsequent
characters to lower-case.

@item @code{reader/upper-case} will have the yielded reader convert to
upper-case all the letters that it reads; again, that is not
sufficient to implement case-insensitivity.

@item @code{reader/debug} causes the generated reader to produce
debugging output.

@end itemize

@node Reader Library, Compatibility and Confinement, Defining a New Reader, Readers
@subsection Reader Library

@cindex reader library
The @code{(system reader)} module exports the @code{default-reader}
procedure which returns a reader equivalent to Guile's built-in
default reader made of re-usable token readers written in C
(@pxref{Token Reader Library}).

@c Include part of API documentation snarffed from `reader-lib.c'.
@include default-reader-doc.texi

Additionally, the @code{(system reader library)} module exports a
number of procedures that ease the re-use of readers.

@c Include part of the API documentation snarffed from `reader-lib.c'.
@include make-guile-reader-doc.texi

@c Include part of the API documentation obtained from `library.scm'.
@include reader-lib-doc.texi


@node Compatibility and Confinement,  , Reader Library, Readers
@subsection Compatibility and Confinement

@findex read-options
@findex read-enable
@findex read-disable
@findex read-set!
@findex read-hash-extend
Guile's core read subsystem provides an interface to customize its
reader, namely via the @code{read-options} (@pxref{Scheme Read,,,
guile, Guile Reference Manual}) and @code{read-hash-extend}
(@pxref{Reader Extensions, @code{read-hash-extend},, guile, Guile
Reference Manual}) procedures.

The main problem with this approach is that changing the reader's
options using these procedures has a global effect since there is only
one instance of @code{read}.  Changing the behavior of a single
function at the scale of the whole is not very ``schemey'' and can be
quite harmful. Suppose a module relies on case-insensitivity while
another relies on case-sensitivity.  If one tries to use both modules
at the same time, chances are that at least one of them will not work
as expected.  Risks of conflicts are even higher when
@code{read-hash-extend} is used: imagine a module that uses
DSSSL-style keywords, while another needs SCSH-style block comments.

@cindex confinement
@cindex reader confinement
@findex read-options-interface
In @code{(system reader confinement)}, guile-reader offers an
implementation of @code{read-option-interface} and
@code{read-hash-extend} that allows to @emph{confine} such settings on
a per-module basis.  In order to enable reader confinement, one just
has to do this:

@smalllisp
(use-modules (system reader confinement))
@end smalllisp

Note that this must be done before the suspicious modules are loaded,
that is, typically when your program starts.  This will redefine
@code{read-options-interface} and @code{read-hash-extend} so that any
future modification performed via Guile's built-in reader option
interface will be confined to the calling module.

@findex current-reader
Starting from Guile 1.8.0, @code{current-reader} is a core binding
bound to a fluid whose value should be either @code{#f} or a reader
(i.e., a @code{read}-like procedure).  The value of this fluid
dictates the reader that is to be used by @code{primitive-load} and
its value can be changed dynamically (@pxref{Loading,
@code{current-reader},, guile, Guile Reference Manual}).

The confined variants of @code{read-options-interface} and
@code{read-hash-extend} rely on this feature to make reader
customizations local to the file being loaded.  This way, invocations
of these functions from within a file being loaded by
@code{primitive-load} take effect immediately.

@c @cindex forward-compatibility
@c When using a version of Guile that does not support the
@c @code{current-reader} feature, the @code{(system reader compat)}
@c module defines a new @code{current-reader} core binding and redefines
@c @code{primitive-load} in order to provide compatibility with newer
@c versions of Guile.


@c chapter ----------------------------------------------------------
@node Internals, Concept Index, API Reference, Top
@chapter Internals

In order to not have to trade too much performance for flexibility,
Guile-Reader dynamically compiles code for the readers defined using
GNU @emph{lightning} (@pxref{Overview, Introduction to GNU
@emph{lightning},, lightning, Using and Porting GNU
@emph{lightning}}).  As of version 1.2c, GNU @emph{lightning} can
generate code for the PowerPC, SPARC, and IA32 architectures.  For
other platforms, Guile-Reader provides an alternative (slower) C
implementation that does not depend on it.  Using the
@emph{lightning}-generated readers typically provides a 5% performance
improvement over the static C implementation.  However, note that
@emph{lightning}@tie{}2.x is not supported yet.

Re-using token readers written in C, as explained in @xref{Token
Reader Library}, does not imply any additional cost: the underlying C
function will be called directly by the reader, without having to go
through any marshalling/unmarshalling stage.

@findex make-guile-reader
Additionally, on the C side, token readers may be initialized
@emph{statically} (except, obviously, token readers made out of a
dynamically-compiled reader).  Making good use of it can improve the
startup time of a program.  For example, @code{make-guile-reader}
(@pxref{Reader Library}) is implemented in C and it uses statically
initialized arrays of token readers.  It still needs to invoke
@code{scm_c_make_reader ()}, but at least, token readers themselves
are ``ready to use''.

Scanners as generated by Flex or similar tools should theoretically be
able to provide better performance because the input reading and
pattern matching loop is self-contained, may fit in cache, and only
has to perform function calls once a pattern has been fully
recognized.


@c *********************************************************************
@node Concept Index, Function Index, Internals, Top
@unnumbered Concept Index
@printindex cp

@node Function Index,  , Concept Index, Top
@unnumbered Function Index
@printindex fn


@bye

@ignore
;;; arch-tag: e82c9dff-b040-4398-a1e9-1e2570cfc025
@end ignore

@c Local Variables:
@c ispell-local-dictionary: "american"
@c End:
