%
% $Header: /home/cvs/root/haskell-report/report/lexemes.verb,v 1.14 2003/01/13 13:08:55 simonpj Exp $
%
%*section 2
%**<title>Haskell 98 Lexical Structure</title>
%**~header
\section{Lexical Structure}\index{lexical structure}
\label{lexical-structure}

% this quote is not in because: it is supposed to show Curry as an
% early advocate of abstract syntax.  However, as far as I know (which
% I can't be sure w/out going to the library), this quote is from 1972
% (when Vol II was published ?), which isn't particularly early.  This
% doesn't seem the sort of thing to get wrong!  If someone wants to do
% the checking and put it in (and re-BibTeX and re-check the page
% breaks :-), please feel free.

%% \begin{quote}
%% ``{\em The various systems of combinatory logic are presented in this
%% book as abstract obs systems.  It was mentioned in \S{}~1 that any
%% such system can be represented as a concrete syntactical system, and
%% that this can be done mechanically.  Nevertheless it seems expedient
%% to give such a representation explicitly for two reasons: in the first
%% place there is at present an almost universal insistence on such a
%% representation; in the second place the discussion of certain
%% philosophical questions is made somewhat easier by having a specific
%% representation before us.}''
%% \begin{flushright}
%% Haskell B.~Curry {\em et al.}\\
%% in {\em Combinatory Logic}, Vol.~II, page 8 \cite{curry-etal:volII}.
%% \end{flushright}
%% \end{quote}

\noindent
In this chapter, 
we describe the low-level lexical structure of \Haskell{}.
Most of the details may be skipped in a first reading of
the report.

\subsection{Notational Conventions}
\label{notational-conventions}

These notational conventions are used for presenting syntax:

\[\ba{cl}
"[pattern]"		& \tr{optional} \\
"\{pattern\}"		& \tr{zero or more repetitions} \\
"(pattern)"		& \tr{grouping} \\
"pat_1 | pat_2"		& \tr{choice} \\
"pat_{\langle{}pat'\rangle{}}"	& \tr{difference---elements generated by "pat"} \\
			& \tr{except those generated by "pat'"} \\
"@fibonacci@"		& \tr{terminal syntax in typewriter font}
\ea\]

Because the syntax in this section describes {\em lexical} syntax, all
whitespace is expressed explicitly; there is no
implicit space between juxtaposed symbols.  BNF-like syntax is used
throughout, with productions having the form:
@@@
nonterm		-> alt_1 | alt_2 | ... | alt_n
@@@

Care must be taken in distinguishing metalogical syntax such as "|"
and "[...]" from concrete terminal syntax (given in typewriter font)
such as @|@ and @[...]@, although usually the context makes the
distinction clear.

\Haskell{} uses the Unicode \cite{unicode} character set.
\index{Unicode character set}
However, source 
programs are currently biased toward the ASCII character set
\index{ASCII character set} used in earlier versions of \Haskell{}.

%	Removed Sept 02
% Haskell uses a pre-processor to convert non-Unicode character sets into
% Unicode.  This pre-processor converts all characters to Unicode and
% uses the escape sequence @\u@"hhhh", where the @"h"@ are hex digits,
% to denote escaped Unicode characters.  Since this translation occurs
% before the program is compiled, escaped Unicode characters may appear in
% identifiers and any other place in the program.

This syntax depends on properties of the Unicode characters as defined
by the Unicode consortium. 
\Haskell{} compilers are expected to make use of
new versions of Unicode as they are made available.

\subsection{Lexical Program Structure}
\label{lexemes}
\label{whitespace}

\input{syntax-lexical}

Lexical analysis should use the ``maximal munch'' rule:
at each point, the longest possible lexeme
satisfying the "lexeme" production is read.
\index{maximal munch rule}
% using a context-independent deterministic lexical analysis
% (i.e.~no lookahead beyond the current character is required).
% Not so -- consider the input "F. 9. 0o 0x 9.0e+f", which 
% should lex as "F . 9 . 0 o 0 x 9.0 e + f" but that needs up 
% to 2 additional characters of lookahead.  
So, although @case@ is a reserved word, @cases@ is not.
Similarly, although @=@ is reserved, @==@ and @~=@ are
not.  

Any kind of "whitespace" is also a proper delimiter for lexemes.

Characters not in the category "ANY" are not valid
in \Haskell{} programs and should result in a lexing error.

\subsection{Comments}

Comments\index{comment} are valid whitespace.

An ordinary comment\index{comment!end-of-line} begins with a sequence of
two or more consecutive dashes (e.g. @--@) and extends to the following newline.
{\em The sequence of dashes must not form part of a legal lexeme.}
For example, ``@-->@'' or ``@|--@'' do {\em not} begin
a comment, because both of these are legal lexemes; however ``@--foo@''
does start a comment.

A nested comment\index{comment!nested} begins with ``@{-@'' 
and ends with ``@-}@''.  No legal lexeme starts with ``@{-@''; 
hence, for example, ``@{---@'' starts a nested comment despite the trailing dashes.

The comment itself is not lexically analysed.  Instead, the first
unmatched occurrence of the string ``@-}@'' terminates the nested
comment.  Nested comments may be nested to any depth: any occurrence
of the string ``@{-@'' within the nested comment starts a new nested
comment, terminated by ``@-}@''.  Within a nested comment, each
``@{-@'' is matched by a corresponding occurrence of ``@-}@''.

In an ordinary comment, the character
sequences ``@{-@'' and ``@-}@'' have no special significance, and, in a
nested comment, a sequence of dashes has no special significance.

Nested comments are also used for compiler pragmas, as explained in
Chapter~\ref{pragmas}.

If some code is commented out using a nested comment, then any
occurrence of @{-@ or @-}@ within a string or within an end-of-line
comment in that code will interfere with the nested comments.

\subsection{Identifiers and Operators}\index{identifier}\index{operator}
\label{ids}

@@@
varid	-> (small \{small | large | digit | @'@ \})_{\langle{}reservedid\rangle{}}
conid		-> large \{small | large | digit | @'@ \} 
reservedid -> @case@ | @class@ | @data@ | @default@ | @deriving@ | @do@ | @else@
	| @if@ | @import@ | @in@ | @infix@ | @infixl@ | @infixr@ | @instance@
	| @let@ | @module@ | @newtype@ | @of@ | @then@ | @type@ | @where@ | @_@
@@@
\indexsyn{varid}%
\indexsyn{conid}%
\indexsyn{reservedid}%

An identifier consists of a letter followed by zero or more letters,
digits, underscores, and single quotes.  Identifiers are lexically
distinguished into two namespaces (Section~\ref{namespaces}): those that begin with a lower-case letter
(variable identifiers) and those that begin with an upper-case letter
(constructor identifiers).  Identifiers are case sensitive: @name@,
@naMe@, and @Name@ are three distinct identifiers (the first two are
variable identifiers, the last is a constructor identifier).

Underscore, ``@_@'', is treated as a lower-case letter, and can occur
wherever a lower-case letter can.  However, ``@_@'' all by itself is a
reserved identifier, used as wild card in patterns.  Compilers that offer
warnings for unused identifiers are encouraged to suppress such warnings for
identifiers beginning with underscore.  This allows programmers to use
``@_foo@'' for a parameter that they expect to be unused.

@@@
varsym		-> ( symbol \{symbol | @:@\} )_{\langle{}reservedop | dashes\rangle{}}
consym		-> (@:@ \{symbol | @:@\})_{\langle{}reservedop\rangle{}}
reservedop	-> @..@ | @:@ | @::@ | @=@ | @\@ | @|@ | @<-@ | @->@ | \verb+@@+ | @~@ | @=>@
@@@
\indexsyn{varsym}%
\indexsyn{consym}%
\indexsyn{reservedop}%

{\em Operator symbols} \index{operator} 
are formed from one or more symbol characters, as
defined above, and are lexically distinguished into two namespaces 
(Section~\ref{namespaces}):
\begin{itemize}
\item An operator symbol starting with a colon is a constructor.
\item An operator symbol starting with any other character is an ordinary identifier.
\end{itemize}
Notice that a colon by itself, ``@:@'', is reserved solely for use
as the Haskell list constructor; this makes its treatment uniform with
other parts of list syntax, such as ``@[]@'' and ``@[a,b]@''.

Other than the special syntax for prefix negation, all operators are
infix, although each infix operator can be used in a {\em
section}\index{section} to yield partially applied operators (see
Section~\ref{sections}).
All of the standard infix operators are just
predefined symbols and may be rebound.  

In the remainder of the report six different kinds of 
names\index{namespaces} will be used:

@@@
varid		       && (\tr{variables})
conid		       && (\tr{constructors})
tyvar	->  varid	& (\tr{type variables})
tycon	->  conid 	& (\tr{type constructors})
tycls	->  conid 	& (\tr{type classes})
modid   ->  conid	& (\tr{modules})
@@@
\indexsyn{varid}%
\indexsyn{conid}%
\indexsyn{tyvar}%
\indexsyn{tycon}%
\indexsyn{tycls}%
\indexsyn{modid}%

Variables and type variables are represented by identifiers beginning
with small letters, and the other four by identifiers beginning with
capitals; also, variables and constructors have infix forms, the other
four do not.  Namespaces are also discussed in
Section~\ref{namespaces}.

\index{qualified name}
A name may optionally be {\em qualified} in certain
circumstances by prepending them with a module identifier.  This
applies to variable, constructor, type constructor and type class
names, but not type variables or module names.  Qualified
names are discussed in detail in Chapter~\ref{modules}.
@@@
qvarid   -> [modid @.@] varid
qconid   -> [modid @.@] conid
qtycon   -> [modid @.@] tycon
qtycls   -> [modid @.@] tycls
qvarsym  -> [modid @.@] varsym
qconsym  -> [modid @.@] consym
@@@
\indexsyn{qvarid}%
\indexsyn{qconid}%
\indexsyn{qtycon}%
\indexsyn{qtycls}%
\indexsyn{qvarsym}%
\indexsyn{qconsym}%
Since a qualified name is a lexeme, no spaces are
allowed between the qualifier and the name.
Sample lexical analyses are shown below.
\[\bto{|l|l|}
\hline
This                       	    & Lexes as this                       \\
\hline
@f.g@                               & @f . g@ (three tokens)             \\
@F.g@		                    & @F.g@ (qualified `g')            \\
@f..@		                    & @f ..@ (two tokens)    \\
@F..@	                            & @F..@ (qualified `.')	    \\
@F.@                                & @F .@ (two tokens)                     \\
\hline\eto\]
The qualifier does not change the syntactic treatment of a name;
for example, @Prelude.+@ is an infix operator with the same fixity as the 
definition of @+@ in the Prelude (Section~\ref{fixity}).


\subsection{Numeric Literals}\index{number!literal syntax}
\label{lexemes-numeric}

@@@
decimal		-> digit\{digit\}
octal		-> octit\{octit\}
hexadecimal	-> hexit\{hexit\}
@@@
\indexsyn{decimal}%
\indexsyn{octal}%
\indexsyn{hexadecimal}%
@@@
integer		-> decimal
                |  @0o@ octal | @0O@ octal
                |  @0x@ hexadecimal | @0X@ hexadecimal


float		-> decimal @.@ decimal [exponent]
	        |  decimal exponent

exponent	-> (@e@ | @E@) [@+@ | @-@] decimal
@@@
\indexsyn{integer}%
\indexsyn{float}%
There are two distinct kinds of numeric literals: integer and
floating.  Integer literals may be given in decimal (the default),
octal (prefixed by @0o@ or @0O@) or hexadecimal notation (prefixed by
@0x@ or @0X@).
Floating literals are always decimal.
A floating literal must contain digits both before and after the
decimal point; this ensures that a decimal point cannot be mistaken
for another use of the dot character.  Negative numeric literals are
discussed in Section~\ref{operators}.  The typing of numeric literals
is discussed in Section~\ref{numeric-literals}.

\subsection{Character and String Literals}
\index{character!literal syntax}
\index{string!literal syntax}
\label{lexemes-char}

@@@
char    ->  @'@ (graphic_{\langle{}@'@ | @\@\rangle{}} | space | escape_{\langle{}@\&@\rangle{}}) @'@
string  ->  @"@ \{graphic_{\langle{}@"@  | @\@\rangle{}} | space | escape | gap\} @"@
escape  ->  @\@ ( charesc | ascii | decimal | @o@ octal | @x@ hexadecimal )
charesc -> @a@ | @b@ | @f@ | @n@ | @r@ | @t@ | @v@ | @\@ | @"@ | @'@ | @&@
ascii   -> @^@cntrl | @NUL@ | @SOH@ | @STX@ | @ETX@ | @EOT@ | @ENQ@ | @ACK@ 
       | @BEL@ | @BS@ | @HT@ | @LF@ | @VT@ | @FF@ | @CR@ | @SO@ | @SI@ | @DLE@ 
       | @DC1@ | @DC2@ | @DC3@ | @DC4@ | @NAK@ | @SYN@ | @ETB@ | @CAN@ 
       | @EM@ | @SUB@ | @ESC@ | @FS@ | @GS@ | @RS@ | @US@ | @SP@ | @DEL@
cntrl   -> ascLarge | @@ | @[@ | @\@ | @]@ | @^@ | @_@
gap     ->  @\@ whitechar \{whitechar\} @\@
@@@
\indexsyn{char}%
\indexsyn{string}%
\indexsyn{escape}%
\indexsyn{charesc}%
\indexsyn{ascii}%
\indexsyn{cntrl}%
\indexsyn{gap}%
\index{\\a@@{\tt {\char'134}a}}%
\index{\\b@@{\tt {\char'134}b}}%
\index{\\f@@{\tt {\char'134}f}}%
\index{\\n@@{\tt {\char'134}n}}%
\index{\\r@@{\tt {\char'134}r}}%
\index{\\t@@{\tt {\char'134}t}}%
\index{\\v@@{\tt {\char'134}v}}%
\index{\\\&@@{\tt {\char'134}\&}}%

Character literals are written between single quotes, as in
@'a'@, and strings between double quotes, as in @"Hello"@.

Escape codes may be used in characters and strings to represent
special characters.  Note that a single quote~@'@ may be used in a string, but
must be escaped in a character; similarly, a double quote~@"@ may be used in a
character, but must be escaped in a string.  @\@ must always be
escaped.  The category "charesc" also includes portable
representations for the characters ``alert'' (@\a@), ``backspace''
(@\b@), ``form feed'' (@\f@), ``new line'' (@\n@), ``carriage return''
(@\r@), ``horizontal tab'' (@\t@), and ``vertical tab'' (@\v@).

Escape characters for the Unicode\index{Unicode character set} character
set, including
control characters such as @\^X@, are also provided.
Numeric escapes such as @\137@ are used to designate the character
with decimal representation 137; octal
(e.g.~@\o137@) and hexadecimal (e.g.~@\x37@) representations are also
allowed.  
% Numeric escapes that are out-of-range of the Unicode standard
% (16 bits) are an error.

Consistent with the ``maximal munch'' rule,
\index{maximal munch rule}
numeric escape
characters in strings consist of all consecutive digits and may
be of arbitrary length.  Similarly, the one ambiguous ASCII escape
code, @"\SOH"@, is parsed as a string of length 1.  The escape
character @\&@ is provided as a ``null character'' to allow strings
such as @"\137\&9"@ and @"\SO\&H"@ to be constructed (both of length
two).  Thus @"\&"@ is equivalent to @""@ and the character
@'\&'@\ is disallowed.  Further equivalences of characters
are defined in Section~\ref{characters}.

A string may include a ``gap''---two backslants enclosing
white characters---which is ignored.
This allows one to write long strings on more than one line by writing
a backslant at the end of one line and at the start of the next.  For
example,
\bprog
@
"Here is a backslant \\ as well as \137, \
    \a numeric escape character, and \^X, a control character."
@
\eprogNoSkip

String literals are actually abbreviations for lists of characters
(see Section~\ref{lists}).

\subsection{Layout}\index{layout}
\label{lexemes-layout}

\Haskell{} permits the omission of the braces and semicolons used in several
grammar productions, by
using {\em layout} to convey the same information.  This allows both
layout-sensitive and layout-insensitive styles of coding, which
can be freely mixed within one program.  Because layout is
not required, \Haskell{} programs can be straightforwardly
produced by other programs.
% without worrying about deeply nested layout difficulties.

The effect of layout on the meaning of a Haskell program
can be completely specified by adding
braces and semicolons in places determined by the layout.  The meaning of
this augmented program is now layout insensitive.

Informally stated, the braces and semicolons are inserted as follows.
The layout (or ``off-side'') rule\index{off-side rule} takes effect
whenever the open brace is omitted after the keyword @where@, @let@,
@do@, or
@of@.  When this happens, the indentation of the next lexeme (whether
or not on a new line) is remembered and the omitted open brace is
inserted (the whitespace preceding the lexeme may include comments).
For each subsequent line, if it contains only whitespace or is
indented more, then the previous item is continued (nothing is
inserted); if it is indented the same amount, then a new item begins
(a semicolon is inserted); and if it is indented less, then the
layout list ends (a close brace is inserted).  If the indentation of the 
non-brace lexeme immediately following a @where@, @let@, @do@ or @of@ is less 
than or equal to the current indentation level, then instead of starting 
a layout, an empty list ``@{}@'' is inserted, and layout processing 
occurs for the current level (i.e. insert a semicolon or close brace). 
A close brace is
also inserted whenever the syntactic category containing the
layout list ends; that is, if an illegal lexeme is encountered at
a point where a close brace would be legal, a close brace is inserted.
The layout rule matches only those open braces that it has
inserted; an explicit open brace must be matched by
an explicit close brace.  Within these explicit open braces,
{\em no} layout processing is performed for constructs outside the
braces, even if a line is 
indented to the left of an earlier implicit open brace.

Section~\ref{layout} gives a more precise definition of the layout rules.

Given these rules, a single newline may actually terminate several
layout lists.  Also, these rules permit:
\bprog
@
f x = let a = 1; b = 2 
          g y = exp2
       in exp1
@
\eprog
making @a@, @b@ and @g@ all part of the same layout
list.

As an example, Figure~\ref{layout-before} shows a (somewhat contrived)
module and Figure~\ref{layout-after} shows the result of applying the
layout rule to it.  Note in particular: (a)~the line beginning @}};pop@,
where the termination of the previous line invokes three applications
of the layout rule, corresponding to the depth (3) of the nested
@where@ clauses, (b)~the close braces in the @where@ clause nested
within the tuple and @case@ expression, inserted because the end of the
tuple was detected, and (c)~the close brace at the very end, inserted
because of the column 0 indentation of the end-of-file token.

\begin{figure}
\outlinec{\small
@
module AStack( Stack, push, pop, top, size ) where
data Stack a = Empty 
             | MkStack a (Stack a)

push :: a -> Stack a -> Stack a
push x s = MkStack x s

size :: Stack a -> Int
size s = length (stkToLst s)  where
           stkToLst  Empty         = []
           stkToLst (MkStack x s)  = x:xs where xs = stkToLst s

pop :: Stack a -> (a, Stack a)
pop (MkStack x s)
  = (x, case s of r -> i r where i x = x) -- (pop Empty) is an error

top :: Stack a -> a
top (MkStack x s) = x                     -- (top Empty) is an error
@
}
%**<div align=center> <h4>Figure 1</h4> </div>
\ecaption{A sample program}
\label{layout-before}
\outlinec{\small
@
module AStack( Stack, push, pop, top, size ) where
{data Stack a = Empty 
             | MkStack a (Stack a)

;push :: a -> Stack a -> Stack a
;push x s = MkStack x s

;size :: Stack a -> Int
;size s = length (stkToLst s)  where
           {stkToLst  Empty         = []
           ;stkToLst (MkStack x s)  = x:xs where {xs = stkToLst s

}};pop :: Stack a -> (a, Stack a)
;pop (MkStack x s)
  = (x, case s of {r -> i r where {i x = x}}) -- (pop Empty) is an error

;top :: Stack a -> a
;top (MkStack x s) = x                        -- (top Empty) is an error
}
@
}
%**<div align=center> <h4>Figure 2</h4> </div>
\ecaption{Sample program with layout expanded}
\label{layout-after}

\end{figure}


%**~footer

% Local Variables: 
% mode: latex
% End:

