\input texinfo
@setfilename dejagnu.info
@c
@c This file documents the GNU Testing Framework ``DejaGnu''
@c
@c Copyright (c) 1992, 1993 Free Software Foundation, Inc.
@c
@c This text may be freely distributed under the terms of the GNU
@c General Public License.
@c

@c FIXME---MAIN TODO LIST!
@c
@c * More examples.
@c 
@c * More index entries.
@c 
@c * Revisit organization.
@c 
@c * discuss Tcl/expect basics---enough to get started (see seminar notes).


@ifinfo
@format
START-INFO-DIR-ENTRY
* DejaGnu: (dejagnu).		The GNU testing framework.
END-INFO-DIR-ENTRY
@end format
@end ifinfo

@syncodeindex ky cp
@syncodeindex fn cp

@c @smallbook
@c @cropmarks

@setchapternewpage odd
@settitle DejaGnu Testing Framework
@titlepage
@finalout
@title The DejaGnu Testing Framework
@subtitle for DejaGnu Version 0.9
@sp 1
@subtitle Jan 1993
@author Rob Savoye
@author Cygnus Support
@page

@tex
{\parskip=0pt \hfill Cygnus Support\par
\hfill edited by Roland H. Pesch\par}
@end tex

@vskip 0pt plus 1filll
Copyright @copyright{} 1992, 1993 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.

Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions.
@end titlepage

@ifinfo
Copyright @copyright{} 1992, 1993 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.

@ignore
Permission is granted to process this file through TeX and print the
results, provided the printed document carries a copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).

@end ignore

Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions.

@node Top
@top DejaGnu

DejaGnu is a framework for running test suites on software tools.

@menu
* Overview::			What is DejaGnu?
* Invoking runtest::		Running the tests
* Configuration::		Configuring DejaGnu
* Internals::			The DejaGnu implementation
* Tests::			How to write a test case
* Extending::			Extending DejaGnu
* Index::			Index
@end menu
@end ifinfo

@node Overview
@chapter What is DejaGnu?
@cindex overview

DejaGnu is a framework for testing other programs.  Its purpose is to
provide a single front end for all tests.  Beyond this, DejaGnu offers
several advantages for testing:

@enumerate
@item
The flexibility and consistency of the DejaGnu framework make it easy
to write tests for any program.

@item
DejaGnu provides a layer of abstraction which makes all tests (if
correctly written) portable to any host or target where a program must
be tested.  For instance, a test for GDB can run (from any Unix based
host) on any target architecture supported by DejaGnu. Currently DejaGnu
runs tests on several single board computers, whose operating software
ranges from just a boot monitor to a full-fledged, Unix-like realtime
OS.

@item
All tests have the same output format.  This makes it easy to integrate
testing into other software development processes.
@end enumerate

DejaGnu is written in @code{expect}, which in turn uses Tcl (Tool
command language).  The framework comprises two parts: the testing
framework and the test suites themselves. Tests are usually written in
@code{expect} using Tcl. Using a Tcl script, DejaGnu can also run
a test suite that is not based on @code{expect}.

@menu
* Basic operation::		A quick guide to using DejaGnu
* Sample test::                 What does a DejaGnu test look like?
* Design goals::		Goals behind DejaGnu
* Future directions::		Where is DejaGnu going
* Tcl and expect::              Reading more about Tcl and Expect
@end menu

@node Basic operation
@section Basic Operation
@cindex basics

Use the @code{runtest} command to run existing DejaGnu tests.  You must
use at least one command-line option: @samp{--tool}, to specify which
program to test.@footnote{@samp{--tool} selects a particular suite of
tests, @emph{not} the name of the executable program to test.
@xref{Site specific,,Site configuration},
for information on the variables that you can use to specify the names
of programs to run.}  (If you always test the same tool, you can record
this option in the file @file{site.exp}, rather than typing it each
time.  @xref{Site specific,,Site Configuration}.)  For example, to run
DejaGnu on @sc{gdb} type (on a @dfn{native} configuration, when the
tests run on the same computer as @code{runtest}):

@example
runtest --tool gdb
@end example

@noindent
When you have a @dfn{cross} configuration, the tests run on a different
computer controlled by the host running @code{runtest}.  In this
situation, you need another option to specify the network address for
the other computer:

@smallexample
runtest --tool gdb --target vx9.munist.com
@end smallexample

By default, @code{runtest} prints only the names of the tests it runs,
the tests that fail (or succeed unexpectedly), and a summary.  To
include all tests in the output (those that pass, and those that are
expected to fail), use the @samp{--all} flag.  For more verbose output
about processes being run, communication, and so on, use
@samp{--verbose}. To see even more output, use multiple @samp{--verbose}
options. The output of the tests goes into two directories,
@file{summary} and @file{detail}.  They will be created if they do not
exist.  @xref{Output files,,The files DejaGnu writes}.

@node Sample test
@section What does a DejaGnu test look like?

@cindex example
Each DejaGnu test is an @code{expect} script; the tests vary widely in
complexity, depending on the nature of the tool and the feature tested.

@kindex gdb.t00/echo.exp
@kindex echo.exp
Here is a very simple @sc{gdb} test---one of the simplest tests shipped
with DejaGnu (extracted from @file{gdb.t00/echo.exp}):@footnote{This is
also one of the first DejaGnu tests created for @sc{gdb}.  More recent
tests use the @samp{gdb_test} procedure, which encapsulates the common
pattern of sending a command to @sc{gdb}, looking for a particular
return text, and testing for common failure conditions.  Using that
procedure, you can write a test equivalent to @samp{echo.exp} simply as
@samp{ gdb_test "echo Hello world!" "Hello world!" }}

@example
# send a string to the GDB stdin:
send "echo Hello world!\n"

# inspect the GDB stdout for the correct reply,
# and determine whether the test passes or fails:
expect @{
	-re "Hello world.*$prompt $"	@{ pass "Echo test" @}
	-re "$prompt $"			@{ fail "Echo test" @}
	timeout				@{ fail "(timeout) Echo test" @}
	@}
@end example

Though brief, this example is a complete test.  It illustrates some of
the main features of DejaGnu test scripts:

@itemize @bullet
@item
The test case does not start the tested program (@sc{gdb} in this case);
all test scripts for interactive tools can assume the corresponding tool
is running.

@item
Comments start with @samp{#}.

@item
The main commands you use to control the tested program are @code{send}
(to give it commands) and @code{expect} (to analyze its responses).

@item
The @code{expect} command uses a list of pairs; a pattern (regular
expression if @samp{-re} specified), followed by an action to run if the
pattern matches output from the program.  Only the action for the
@emph{first} matching pattern will execute.

@item
Test cases use the commands @code{pass} and @code{fail} to record the
test outcome.
@end itemize

@node Design goals
@section Design Goals
@cindex design goals

DejaGnu was designed to create a testing tool that:

@itemize @bullet
@item
is useful to developers while fixing bugs;

@item
automates running many tests during the release process;

@item
is portable between a variety of host computers;

@item
supports cross-development testing;

@item
permits testing interactive programs, like @sc{gdb}; and

@item
permits testing batch oriented programs, like @sc{gcc}.
@end itemize

Some of these requirements proved challenging.  For example, interactive
programs do not lend themselves very well to automated testing.  But all
the requirements are important: for instance, it is imperative to test
the @sc{gdb} back end on several remote targets.

Probably the greatest challenge was cross-development testing (which can
be a real nightmare).  Most cross-development environments are
customized by each developer.  Even when buying packaged boards from
vendors there are many differences.  The communcation interfaces vary
from a serial line to ethernet.  DejaGnu was designed with a modular
communication setup, so that each kind of communication can be added as
required, and supported thereafter.  Once a communication procedure is
coded, it can be used in any test.  Currently DejaGnu can use
@code{rsh}, @code{rlogin}, @code{telnet}, @code{tip}, @code{kermit}, and
@code{mondfe}@footnote{@code{mondfe} is an @sc{amd} specific monitor
program freely available from @sc{amd}.} for remote communications.
@kindex mondfe

@node Future directions
@section Future Directions
@cindex future directions

In the near future, there are two parallel directions for DejaGnu
development.  The first is to add support for more targets; part of this
consists of porting Tcl and @code{expect} to new hosts as well.  The
second is to add more test cases, and support for more tools.  It would be particularly useful to add tests for conformance to the
@sc{ansi} C and C++ specifications, to @sc{POSIX}, and to other
standards.

Another useful extension would be to permit testing programs with a more
complex interface, whether text based or GUI based.  Two useful
components already exist: a Tcl based X window toolkit, and a terminal
package for @code{expect}. Both of these must be merged into DejaGnu in
a way that permits testing programs that run in each environment.

@node Tcl and expect
@section Tcl and Expect
@cindex tool command language
@cindex tcl
@cindex Ousterhout, John K.
Tcl was introduced in a paper by John K. Ousterhout at the 1990 Winter
Usenix conference, @cite{Tcl: An Embeddable Command Language}.  The
paper is included in PostScript form in the @file{doc} subdirectory of
the Tcl distribution.  The @code{man} page @cite{Tcl(3)} contains even
more detail.

@cindex @code{expect} scripting language
@cindex Libes, Don
Don Libes introduced @code{expect} in his paper @cite{expect: Curing
Those Uncontrollable Fits of Interaction} at the 1990 Summer Usenix
conference.  The paper is included in PostScript form in the
@code{expect} distribution (as are several other papers about
@code{expect}).  See also the @code{man} page @cite{expect(1)}.

@node Invoking runtest
@chapter Running the Tests
@cindex invoking
@cindex running
@cindex command line options
@cindex options
@kindex runtest

You can specify two kinds of things on the @code{runtest} command line:
command line options, and environment variables for the test scripts.
The options are listed alphabetically below.  The defaults for
all @code{runtest} options are under your control; see @ref{Site
specific,,Site configuration}.

@cindex exit code from @code{runtest}
@cindex @code{runtest} exit code
@code{runtest} returns an exit code of @code{1} if any test
has an unexpected result; otherwise (if all tests pass or fail as
expected) it returns @code{0} as the exit code.

DejaGnu flags the outcome of each test as one of these four cases:

@table @code
@item PASS
The most desirable outcome: the test succeeded, and was expected to
succeed.

@item XPASS
A pleasant kind of failure: a test was expected to fail, but succeeded.
This may indicate progress; inspect the test case to determine whether
you should amend it to stop expecting failure.

@item FAIL
A test failed, although it was expected to succeed.  This may indicate
regress; inspect the test case and the failing software to locate the bug.

@item XFAIL
A test failed, but it was expected to fail.  This result indicates no
change in a known bug.
@end table

DejaGnu may also have the following messages:

@table @code
@item ERROR
Indicates a major problem in running the test. This is usually an
unrecoverable error, such as a missing file or loss of communication to the
target.

@item WARNING
Indicates a possible problem in running the test. Usually warnings
correspond to recoverable errors, or display an important message about
the following tests.
@end table

@example
runtest @var{envvar}=@var{value}@dots{}
[ --all ]  [ --connect @var{type} ]  [ --debug ]  [ --diff ]
[ -h | --help ]  [ --mail "@var{name1} @var{name2} @dots{}" ]
[ --noreboot ]  [ --outdir @var{path} ]
[ --runtests "@var{file1} @var{file2}@dots{}" ]  [ --srcdir @var{path} ]
[ --strace @var{n} ]  [ --target @var{name} ]  --tool @var{tool}
[ -v | --verbose ]  [ -V | -version ]
@end example

@table @code
@item @var{envvar}=@var{value}
You can define environment variables for use by your test scripts in the
same style used with @code{make}.  For example, @samp{runtest
GDB=gdb.old} defines a variable called @samp{GDB}; when your scripts
refer to @samp{$GDB} in this run, they use the value @samp{gdb.old}.

The default environment variables used for most tools are defined in the
main DejaGnu @code{Makefile}; their values are captured in the
@file{site.exp} file.  @xref{Site specific,,Site configuration}.

@item --all
@cindex test output
Display all test output.  By default, @code{runtest} shows only the
output of tests that produce unexpected results; that is, tests with
status @samp{FAIL} (unexpected failure), @samp{XPASS} (unexpected
success), or @samp{ERROR} (a severe error in the test case itself).
Specify @samp{--all} to see output for tests with status @samp{PASS}
(success, as expected) @samp{XFAIL} (failure, as expected), or
@samp{WARNING} (minor error in the test case itself).

@item --baud @var{baud rate}
@itemx -b
@cindex baud rate
Set the default baud rate to something other than 9600.  (Some serial
interface programs, like @code{tip}, use a separate initialization file
instead of this value.)

@item --connect @var{type}
@cindex connecting to target
How to connect to the target testing environment, if the target is not
the computer running @code{runtest}.  For example, use @samp{--connect}
to change the program used to connect to a ``bare board'' boot monitor.
The choices available in the DejaGnu distribution are @samp{rlogin},
@samp{telnet}, @samp{rsh}, @samp{tip}, @samp{kermit}, and
@samp{mondfe}.@footnote{@code{mondfe} is specific to AMD targets using
the UDI protocol; it will not work for other targets.}

@noindent
The default for this option depends on the configured target
(@pxref{Cross targets,,Remote targets supported}).

@item --debug
@cindex debugging test cases
Turns on the @code{expect} internal debugging output.  Debugging output
is displayed as part of the @code{runtest} output, and logged to a file
called @file{dbg.log}.  The output shows all attempts at matching the
test output of the tool with the scripted patterns describing expected
output.  The output generated with @samp{--strace} also goes into this
file.

@item --diff
@itemx --diff -@var{flags}
@emph{Warning: this option is not yet implemented.}
Show on the standard output device the differences (using @code{diff})
between the summary output from the current test run, and the summary
from the previous run.  You can optionally specify @code{diff}
flags (e.g. to select an alternate format) immediately after the
@samp{--diff} option.
@ignore
@c FIXME! Enable when --diff implemented (and check it works this way)!
Without @samp{--diff}, the differences are saved in the @file{summary}
directory in a file called @file{@var{toolname}-diff.txt}.
@end ignore
@c FIXME! Is toolname-diff.txt saved even if also displayed with --diff?
@c FIXME! Orig txt spoke of renaming last summary output, but didn't say
@c what it was renamed *to*.  Assumed renamed only long enough for diff,
@c then erased; if kept, say WHAT OLD SUMMARY IS RENAMED TO.
@c (cannot resolve these for certain till Rob or someone implements option).

@item --help
@itemx -h
@cindex help with @code{runtest}
Prints out a short summary of the @code{runtest} options, then exits
(even if you also specify other options).

@item --mail "@var{name1} @var{name2} @dots{}"
@itemx --mail @var{name}
@cindex emailing results
@cindex mailing results
@emph{Warning: this option is not yet implemented.}
Send the testing output to the specified list, via email.  If used with
the @samp{--diff} option, only the diff file is mailed.  The quotes are
only needed if you supply more than one name with a single @samp{--mail}
option.
@c FIXME! What output mailed if *not* w/--diff?  Summary only, or
@c everything on stdout, or...?
@c (revisit after option implemented)

@item --noreboot
@cindex rebooting the target
@emph{Do not} reboot the target board when @code{runtest} initializes.
Usually, when running tests on a separate target board, it is safer to
reboot the target to be certain of its state.  However, when developing
test scripts, this option can save a lot of time.

@item --outdir @var{path}
@cindex output directory
The name of a directory where output logs are saved (the default is
@samp{.}, the directory where you start @code{runtest}).  If requested,
the @file{dbg.log} debugging log goes in this directory; the ordinary
logs are placed in two subdirectories.  @file{summary}, which holds the
summary logs; and @file{detail}, which holds the full output of the tool
being tested.  You must create the @var{path} subdirectory before
calling @code{runtest}.

@item --runtests "@var{file1} @var{file2} @dots{}"
@itemx --runtests @var{filename}
@cindex selecting a range of tests
Specify the names of test files to run.  Use this option to choose a
subset of the tests available in the @samp{--srcdir} directory.  You
may supply just filenames with no path; the subdirectory name is not
necessary.
@c FIXME! Rob may add wildcards or regexps; find out which, doc if so.

@item --srcdir @var{path}
@cindex source directory
The top directory for test scripts to run. @code{runtest} will look in this
directory for any subdirectory whose name begins with the toolname.  For
instance, when testing @sc{gdb}, @code{runtest} will run tests from
subdirectories @samp{gdb*} (with the usual shell-like filename
expansion).  If you do not use @samp{--srcdir}, @code{runtest} looks for
test directories under the directory that contains the DejaGnu source.

@item --strace @var{n}
@cindex tracing Tcl commands
Turns on internal tracing for @code{expect}, to @var{n} levels deep. By
adjusting the level, you can control the extent to which your output
expands multi-level Tcl statements.  This way output can ignore some
levels of @code{case} or @code{if} statements.  Each procedure call or
control structure counts as one ``level''.

The output is recorded in the same file, @file{dbg.log}, used for output
from @samp{--debug}.

@item --target @var{name}
@cindex specifying target
The name of the testing target machine.  If the testing target has
IP network support (for example, @code{RPC} or @code{NFS}), this is the
network name for the target itself.  (This is @emph{not the
configuration string} you specify as a target with @code{configure}; the
@code{runtest} option names a particular target, rather than describing
a class of targets.)  For targets that connect in other ways, the
@var{name} string depends on the connection method.  @xref{Cross
targets,,Remote targets supported}.

@item --tool @var{tool}
@cindex selecting tests for a tool
Specify the tool (program) you are testing.  @var{tool} is used for two
purposes: to identify the right initialization module, and (as a prefix)
to identify subdirectories containing the desired tests.
It is @emph{not} used to name the executable program to test;
executable tool names (and paths) are recorded in @file{site.exp}
(@pxref{Site specific,,Site configuration}).

@item --verbose
@itemx -v
@cindex turning on output
@cindex output, turning on
Turns on more output.  Repeating this flag increases the level of
verbosity.  Level one (@samp{-v}) is straight test output. Level two
(@samp{-v -v}) is messages on options, configuration, and process
control. Verbose messages are not logged; their purpose is to help debug
the framework or a test script.

@item --version
@itemx -V
@cindex version numbers
Prints out the versions of DejaGnu, @code{expect} and Tcl.
@end table

@node Configuration
@chapter Configuring DejaGnu
@cindex configuring DejaGnu

Once you have the DejaGnu source unpacked and available, the first
requirement to start using DejaGnu is to configure for the host (where
@code{runtest} will execute) and target (where the tools---or their
output---will be tested).  Moreover, it is usually best to configure in
a directory separate from the source tree, using the @samp{--srcdir}
option to @code{configure} to specify where the source is located:

@example
configure @var{host-type} --target=@var{target-type} --srcdir=@var{path}
@end example

@cindex native configuration
To run tests on a full-fledged operating system, you can specify
host and target as the same machine; we call this a @dfn{native}
configuration.  For native configurations, specifying the target
explicitly is not necessary.  For example, to configure for a Sun-4
running SunOS:

@example
configure sun4 --srcdir=/usr/cygnus/src/deja-gnu
@end example

@cindex cross configuration
When the host and target are not the same (typically when testing
software tools on a ``bare board'' controlled from a separate
general-purpose computer), we refer to a @dfn{cross}
configuration.  For example, to run tests controlled by Sun-4 on an
@sc{amd} 29000 board, communicating via the @sc{amd} ``UDI'' protocol:

@example
configure sun4 --target=a29k-amd-udi --srcdir=/usr/cygnus/src/deja-gnu
@end example

DejaGnu uses Cygnus Configure.  @xref{Top,,What Configure
Does,configure.info,Cygnus Configure}, for more information on its
capabilities, conventions, and command-line options.

Aside from its usual effect of preparing and building the executables on
your host, configuring and making DejaGnu creates a file called
@file{site.exp}.  This file encapsulates information from both
@code{configure} and @code{make}, as default values for DejaGnu tests.
@xref{Site specific,,Site Configuration}.

@file{site.exp} is initially created by @code{configure}, using
@code{make}. The @file{Makefile} generated by @code{configure} also has
a target @code{site.exp} which edits this file to keep it updated.  You
can override the default values from this automatic process in a
user-editable section of the file.

@cindex auxiliary programs
@cindex @code{make} builds part of tests
After configuring the top-level DejaGnu directory, run @code{make} to
build auxiliary programs required by some of the tests.

DejaGnu has some additional requirements for target support, beyond the
general-purpose provisions of Cygnus Configure.  DejaGnu must actively
communicate with the target, rather than simply generating or managing
code for the target architecture.  Therefore, each tool requires an
initialization module for each target.  For new targets, you must supply
a few user-supplied functions to adapt DejaGnu to the target.
This permits DejaGnu itself to remain target independent.  @xref{Init
module,,Platform specific initialization module}.

@menu
* Site specific::		Site specific configuration file
* Init module::			Platform specific initialization module
* Installation::                Installing DejaGnu
* Cross targets::               Remote targets supported
@end menu

@node Site specific
@section Site configuration
@cindex site configuration
@kindex site.exp

The site configuration file, @file{site.exp}, ties the DejaGnu test
scripts into the @code{configure} and @code{make} programs.  This file
is generated by typing @code{make site.exp}. When you run
@code{configure}, it automatically runs @code{make} for this purpose.
@file{site.exp} has two sections, marked by comment text. The first section
starts with:

@example
## these variables are automatically generated by make ##
# Do not edit here. If you wish to override these values
# add them to the last section
@end example

@noindent
Do not edit this first section; if you do, your changes are replaced
next time you run @code{configure}.  The first section ends with this
line:

@example
## All variables above are generated by configure. Do Not Edit ##
@end example

The first section is the part that is generated by @code{make}. It is
essentially a collection of environment variable definitions from the
@file{Makefile}. Since they are generated by @code{make}, they contain
the values as specified by @code{configure}.  You can customize these
values by using the @code{--site=ms-*} option to @code{configure}. In
particular, this section contains the @file{Makefile} variables for host
and target configuration data.

In the second section, you can override any default values (locally to
DejaGnu) for all the variables.  The second section can also contain
your preferred defaults for all the command line options to
@code{runtest}. This allows you to easily customize @code{runtest} for
your preferences in each configured DejaGnu tree, so that you need not
type options repeatedly on the command line.  (The second section may
also be empty, if you do not wish to override any defaults.)

@cindex tool defaults
@cindex configuration dependent defaults
@cindex defaults for tool names
Here are the @file{Makefile} variables set in @file{site.exp}.
@emph{The default values depend on your configuration.}

To override the values captured in the first part of @file{site.exp} by
@code{configure} and @code{make}, simply set your own value for any of
these variables @emph{in the second part} of @file{site.exp}.  Set the
variables using the Tcl @code{set} command; for example, to run tests on
an experimental version of @sc{gcc}, the second part of your
@file{site.exp} might include:

@example
set CC "/working/devo/gcc/gcc"
@end example

@c FIXME! Do any of the defaults still shown depend on config?  
@c        If so, delete. 
@ftable @code
@item CC
@cindex default C compiler 
The C compiler. The default value is @code{"gcc"}.

@item CFLAGS
The list of options passed to the C compiler. The default value
is @code{"-g"}. 

@item CPLUS
@cindex default C++ compiler
The C++ compiler. @code{make} uses @code{C++}, but @code{expect} cannot
use the @samp{+} in a variable name.  The default value is @code{"gcc"}.
@c FIXME!  (1) is this to change to CXX or not?
@c FIXME!  (2) What does it mean that "make uses C++"??  Should that be
@c             Cygnus configure rather than make?

@item CPLUSFLAGS
The list of options for the C++ compiler. The default value is @code{"-g"}.

@item AS
@cindex default assembler
The assembler. The default value is @code{"as"}.

@item ASFLAGS
The list of options for the assembler.  The default value is @code{""}.

@item FC
@cindex default @sc{fortran} compiler
The @sc{fortran} compiler. The default value is @code{""}.

@item FCFLAGS
The list of options for the @sc{fortran} compiler. The default value is
@code{""}.

@item LD
@cindex default linker
The linker. The default value is @code{"ld"}.

@item LDFLAGS
The list of options for the linker. The default value is @code{""}.

@item RANLIB
@cindex default archive randomizer
The program for writing an index to object code archive files.
The default value is @code{"ranlib"}.

@item RANLIBFLAGS
The options for @code{$RANLIB}. The default value is @code{""}.

@item NM
@cindex default symbol lister
The program that dumps the list of symbols from an object file. The
default value is @code{"nm"}. 

@item NMFLAGS
The options for @code{$NM}. The default value
is @code{""}. 

@item SIZE
@cindex default section size dumper
This is the program that lists the size of all the sections in the
compiled file. The default value is @code{"size"}.

@item SIZEFLAGS
The options for @code{$SIZE}. The default value is
@code{""}. 

@item STRIP
@cindex default symbol table stripper
The program to strip the symbol table from compiled files
to reduce size. The default value is @code{"strip"}.

@item STRIPFLAGS
The options for @code{$STRIP}. The default
value is @code{""}. 

@item OBJDUMP
@cindex default @sc{coff} dumper
The program that dumps the parts of a @sc{coff} file. The
default value is @code{"objdump"}. 

@item OBJDUMPFLAGS
The options for the @sc{coff} file dumper. The default value is
@code{""}. 

@item LEX
@cindex default lexical analyzer generator
The program that makes lexical analyzers. The default value is
@code{"flex"}. 

@item LEXFLAGS
The options for the lexical analyzer generator.  The default value is
@code{""}. 

@item YACC
@cindex default parser generator
The ``compiler-compiler'', a program to generate parsers. The
default value is @code{"byacc"}. 

@item YACCFLAGS
The options for @code{$YACC}. The default value
is @code{""}. 

@item AR
@cindex default archiver
The program that builds object code archives. The default value is @code{"ar"}.

@item ARFLAGS
The options for the archiver. The default value is @code{""}.

@item CPP
@cindex default preprocessor
The C compiler preprocessor. The default value is @code{"gcc
-E"}. 

@item CPPFLAGS
The options for the C preprocessor. The default value is @code{""}.

@item GDB
@cindex default debugger
The C and C++ debugger. The default value is @code{"gdb"}.

@item GDBFLAGS
The options for the debugger. The default value is @code{""}.

@item DIFF
@cindex default @code{diff} program
The program that finds differences in text files. The default
value is @code{"diff"}. 

@item DIFFFLAGS
The options for the @code{diff} program. The default value is @code{""}.
@end ftable

@noindent
The following variables, also in @file{site.exp}, extend the standard
set of configuration variables. They should never be redefined. They permit
@code{runtest} to determine its configuration at runtime:

@ftable @code
@item host_os
The host operating system name, as seen by @file{config.sub}.  The
default depends on the value you specify with @samp{configure --host}
for your configuration.

@item host_alias
@cindex host configuration variables
A shorter name for the host system. The default depends on the value you
specify with @samp{configure --host} for your configuration.

@item host_cpu
The host processor type. The default depends on the value you
specify with @samp{configure --host} for your configuration.

@item host_vendor
The manufacturer of the host system. The default depends on the value you
specify with @samp{configure --host} for your configuration.

@item target_os
@cindex target configuration variables
The target operating system name, as seen by @file{config.sub}. The
default depends on the value you specify with @samp{configure --target}
for your configuration.

@item target_alias
A shorter name for the target system. The default depends on the value
you specify with @samp{configure --target} for your configuration.

@item target_cpu
The target processor type. The default depends on the value you specify
with @samp{configure --target} for your configuration.

@item target_vendor
The manufacturer of the host system.  The default depends on the value
you specify with @samp{configure --target} for your configuration.
@end ftable

@cindex option defaults
@cindex @code{runtest} option defaults
@cindex variables for option defaults
@cindex defaults, option
In the user editable second section of @file{site.exp}, you can not only
override the configuration variables captured in the first section, but
also specify default values for all the @code{runtest} command line
options.  Each command line option has an associated Tcl variable; as
for the configuration variables, use the Tcl @code{set} command to
specify a new value.  The following table describes the correspondence
between command line options and variables you can set in
@file{site.exp}.  @xref{Invoking runtest,,Running the Tests}, for 
explanations of the command-line options.

@c FIXME?  Missing: --debug, --help (?), --noreboot, --version (?)

@kindex all_flag
@kindex baud
@kindex connectmode
@kindex diff_files
@kindex mailing_list
@kindex outdir
@kindex runtests
@kindex srcdir
@kindex tracelevel
@kindex targetname
@kindex tool
@kindex verbose

@cindex command line option variables
@cindex Tcl variables for option defaults
@cindex default options, controlling
@cindex options, Tcl variables for defaults

@ifinfo
@display
runtest      Tcl   
option     variable     description
__________ ________     ___________________________________________

--all      all_flag     display all test results if set

--baud     baud         set the default baud rate to something other
                        than 9600.
--connect  connectmode  @samp{rlogin}, @samp{telnet}, @samp{rsh}, 
                        @samp{kermit}, @samp{tip}, or @samp{mondfe}

--diff     diff_files   display @code{diff} of summary reports

--mail     mailing_list address list for mailing test output

--outdir   outdir       directory for @file{summary} and @file{detail}

--runtests runtests     list of tests to run

--srcdir   srcdir       directory to scan for test subdirectories

--strace   tracelevel   a number: Tcl trace depth

--target   targetname   network name of testing target or its host

--tool     tool         name of tool to test; identifies init, test subdir

--verbose  verbose      verbosity level: a number, 0 or greater
@end display
@end ifinfo

@tex
\vbox{\halign{\hfil \tt #\quad &\quad\tt #\hfil &\hbox{\vtop{{\hsize=3in\rm#\hfil\strut}}}\hfill\cr
\cr
{\it runtest}&{\it Tcl}\cr
{\it option}&{\it variable}&{\it description}\cr
\noalign{\hrule width\hsize}\cr
--all     &all\_flag    &display all test results if set\cr
--baud    &baud         &set the default baud rate to something other
                        than 9600.\cr
--connect &connectmode &@samp{rlogin}, @samp{telnet}, @samp{rsh},
                        @samp{kermit}, @samp{tip}, or @samp{mondfe}\cr
--diff    &diff\_files  &display @code{diff} of summary reports\cr
--mail    &mailing\_list&address list for mailing test output\cr
--outdir  &outdir      &directory for @file{summary} and @file{detail}\cr
--runtests&runtests    &list of tests to run\cr
--srcdir  &srcdir      &directory to scan for test subdirectories\cr
--strace  &tracelevel  &a number: Tcl trace depth\cr
--target  &targetname  &network name of testing target or its host\cr
--tool    &tool        &name of tool to test; identifies init, test subdir\cr
--verbose &verbose     &verbosity level: a number, 0 or greater\cr
}}
@end tex

@node Init module
@section Initialization Module
@cindex tool initialization
@cindex setting up targets

The initialization module has two purposes: to provide tool and target
dependent procedures, and to start up the tool to the point where it is
ready to operate.  The latter includes establishing communications with
the target.  All the tests assume that the tool is already running and
communicating.  Initialization modules for non-interactive programs may
only need to supply the support functions.

A separate initialization module is needed for each combination of tool
and target.  In a test running native, writing the Tcl code for
initialization is usually quite simple.  For cross configurations,
however, more elaborate instructions are usually needed to describe how
to talk to a remote target.

@code{runtest} requires four @code{expect} procedures in each
initialization module.  The procedure names all begin with the name of
the tool; for example, the start procedure for @sc{gdb} is called
@code{gdb_start}. 

The procedures are called @code{@var{tool}_start},
@code{@var{tool}_load}, @code{@var{tool}_exit}, and
@code{@var{tool}_version}.  @xref{Target dependent,,Target dependent
functions}, for full descriptions.

@cindex initialization module name
@code{configure} links the appropriate initialization module for a
particular target to the filename @file{@var{tool}_init.exp}; for
example, the appropriate initialization for @sc{gdb} is linked to
@file{gdb-init.exp}.  If you extend DejaGnu, you must amend
@code{configure.in} to arrange this link for your new initialization
module.  The easiest approach is to follow the existing naming
convention: use the name @file{@var{target_os}-@var{tool}} for the
configuration file.  The prefix @var{target_os} is the name of the
target operating system, as captured in the @file{site.exp} variable
@code{target_os}.

@node Installation
@section Installing DejaGnu

@cindex Installing DejaGnu
Type @samp{make install} to install DejaGnu in your filesystem.  This
uses the Makefile value of @code{exec_prefix}, originally an option to
@code{configure}. @samp{make install} does the following things: First,
it looks in @file{$exec_prefix/lib} for a directory called
@file{deja-gnu}. If this directory does not exist, it is created.
Second, all the library files used to support the framework are copied
into this directory.  Third, another directory is created within
@file{deja-gnu}, using @samp{$target_alias} as the name.  Fourth, the
@file{site.exp} file for that configuration is copied into this
directory.

Finally, @samp{make install} copies @file{runtest.exp}, the Tcl code for
the main DejaGnu test driver, to @file{$exec_prefix/bin}. At this time
the path recorded in @file{runtest.exp} for the library files
is changed to reflect their installed location.  @code{runtest} is also
copied to this directory, but its name is changed, because this
installed version passes the configuration string to @file{runtest.exp}
to specify which @file{site.exp} file to use.  The installed name for a
particular configuration is @code{$target_alias-runtest}.

@node Cross targets
@section Remote targets supported

@cindex targets
@cindex remote testing
You can set the target name and the connect mode in the @file{site.exp}
file (using the Tcl variables @samp{targetname} and @samp{connectmode},
respectively), or on the command line (using @samp{--target} and
@samp{--connect}).

The DejaGnu distribution includes support for the following remote
targets.  If you suspect a communication problem, try running the
connection interactively from @code{expect}.@footnote{There are three
ways of running @code{expect} as an interactive interpreter.  You can
use the @samp{expect -i} command-line flag; you can call @code{expect}
with no arguments or options; or you can run the command
@code{interpreter} from any @code{expect} procedure.  Use @code{return}
to get back to the calling procedure (if any), or @code{return -tcl} to
make the calling procedure itself return to its caller; use @code{exit}
or end-of-file to leave @code{expect} altogether.}  Run the program whose name
is recorded in @samp{$connectmode}, with the arguments in
@samp{$targetname}, to establish a connection.  You should at least be
able to get a prompt from any target that is physically connected.
@xref{Adding tools or targets,,Adding a tool or a target}, for some
discussion of writing and debugging new targets.
@ignore
expect -i for interactive run
or just expect with no args
or call cmdn "interpreter" until return (to caller)) or return -tcl (to
caller's caller)
@end ignore

@table @strong
@item @sc{amd} 29000, with UDI protocol
Configure DejaGnu for target @samp{a29k-amd-udi}.  (Cygnus
@code{configure} also recognizes the abbreviation @samp{udi29k}.)  Then,
to run tests, use the @code{runtest} target name to specify whether you
want to use a simulator, or a particular hardware board.  The particular
string to use with @samp{--target} will depend on your UDI setup file,
@file{udi_soc} (if @file{udi_soc} is not in your working directory, the
environment variable @samp{UDICONF} should contain a path to this file).
For example, if your UDI setup file includes these lines:

@smallexample
iss   AF_UNIX  *   isstip -r /home/gnu/29k/src/osboot/sim/osboot
mon   AF_UNIX  *   montip -t serial -baud 9600 -com /dev/ttyb
@end smallexample

@noindent
use @samp{--target iss} to run tests on the simulator, and
@samp{--target mon} to run tests on the 29K hardware.  See the
manufacturer's manuals for more information on UDI and @file{udi_soc}.

The default connect protocol is @samp{mondfe} with either back end.
@code{mondfe} is the only shell DejaGnu supports for UDI targets.

@quotation
@emph{Warning:} This target requires @sc{gdb} version 4.7.2 (or
greater).  Earlier versions of @sc{gdb} do not fully support the
@code{load} command on this target, so DejaGnu has no way to load
executable files from the debugger.
@end quotation

@item Motorola 680x0 boards, a.out or COFF object format
Configure DejaGnu for any remote target matching @samp{m68k-*}.

@quotation
@emph{Warning:} Most @samp{m68k-*} configurations run all tests only for
native testing (when the target is the same as the host).  When you
specify most of these targets for a cross configuration, you will only be
able to use tests that run completely within the host (for example,
tests of the binary utilities such as the archiver; or compiler tests
that only generate code rather than running it).
@end quotation

To run a.out or @sc{coff} binaries on a remote M68K, you must configure
DejaGnu for a particular target board.  @samp{m68k-abug} is an example.
(In general for an embedded environment, because it dosen't have absolute
addresses, a.out is not a good choice for output format in any case; most
often S-records or Hex-32 are used instead.)

@item Motorola 68K MVME board running 135ABug boot monitor
Configure DejaGnu for target @samp{m68k-abug-aout} or
@samp{m68k-abug-coff}.  (@emph{WARNING}: these target name may change in
the near future.)  This boot monitor can only download S-records;
therefore, the DejaGnu tests for this environment require a linker
command script to convert either output format to S-records, setting the
default addresses for @code{.text}, @code{.bss}, and @code{.data}.

With this configuration, the default for @samp{--connect} is @samp{tip};
this is the only supported communications protocol to @samp{m69k-abug-*}
targets.  @samp{tip} uses an @sc{ascii} downloader (the @code{~put}
command) to load S-records into the target board.  The @samp{--target}
string must be a machine name that @code{tip} understands (for example,
on some @code{tip} implementations it must be an entry from the
initialization file for @code{tip}; this file is sometimes called
@file{/etc/remote}).

See your system documentation for information on how to create new
entries in @file{/etc/remote}.  (Some @sc{unix} systems are distributed
with at least one default entry with a name resembling @samp{hardwire};
if your system has one, you can edit it, or make a modified copy with a
new name.)  When you have a working @file{/etc/remote} entry
@var{abugtarget}, you should be able to type @samp{tip
@var{abugtarget}}, and get the prompt @samp{135ABUG>} from the board.
Use the same @var{abugtarget} string with @samp{runtest --target}.

@item VxWorks (Motorola 68K or Intel 960)
Configure DejaGnu for one of the two VxWorks targets: either
@samp{m68k-wrs-vxworks}, or @samp{i960-wrs-vxworks}.  (@code{configure}
also recognizes the abbreviations @samp{vxworks68} or
@samp{vxworks960}.)  Since both targets support IP addressing, specify
the network address (for example, a host name from @file{/etc/hosts})
with @samp{--target}.  

The default connect protocol is @samp{rlogin}, but you can use any of
@samp{--connect rlogin}, @samp{--connect telnet}, or @samp{--connect
rsh}.

Test scripts need no special code to load programs into these targets;
since VxWorks supports NFS, all you must do is ensure test programs are
on an exported filesystem.

@cindex VxWorks, link with @samp{-r}
When you compile for VxWorks, use the linker @samp{-r} option to make
the linker output relocatable---at least if you want to use library
routines. Many standard C routines are included in VxWorks; often no
additional libraries are needed.  See your VxWorks system documentation
for additional details.
@end table

@node Internals
@chapter The DejaGnu Implementation
@cindex operating principles
@cindex internal details

DejaGnu is entirely written in @code{expect}, which uses Tcl as a
command language.  @code{expect} serves as a very programmable shell;
you can run any program, as with the usual Unix command shells---but
once the program is started, your @code{expect} script has fully
programmable control of its input and output.  This does not just apply
to the programs under test; @code{expect} can also run any auxiliary
program, such as @code{diff} or @code{sh}, with full control over its
input and output.

The most important part of DejaGnu---and the largest body of code---is
the suite of test scripts it includes.  Future releases will include
even more tests, developed throughout the free software community.

@code{runtest}, of course, is the glue to tie together and manage the
test scripts.  @code{runtest} itself has these essential functions:

@enumerate
@item
Locating the individual test scripts.  @code{runtest} locates the tests
by exploiting a straightforward naming convention.

@item
Providing an extended test environment, by defining additional Tcl
functions beyond those already in @code{expect}.

@item
Locating target-dependent functions, to standardize the test environment
across a wide variety of test platforms.
@end enumerate

@menu
* Names::                       Conventions for using tool names
* Builtins::			Built in Tcl functions
* Target dependent::		Functions supplied by the init module
* Input files::			The files DejaGnu depends on
* Output files::		The files DejaGnu produces
@end menu

@node Names
@section Conventions for using tool names

DejaGnu uses @samp{$tool}, the name of the tool under test, to tie
together the testing configuration in a straightforward but flexible
way.

Each initialization module is tool-dependent; their names use the tool
name as a prefix.  @xref{Target dependent,,Target dependent functions}.

@samp{$tool} is @emph{not} used to invoke the tool, since sites that run
multiple configurations of a particular tool often call each
configuration by a different name.  @code{runtest} uses the
configuration-dependent variables captured in @file{site.exp} to
determine how to call each tool.

@code{runtest} also uses tool names to find the directories containing
the tests. DejaGnu scans the source directory (specified with
@code{--srcdir}) for all directories whose names start with the tool
name followed by a period. For instance, directories that start with
@samp{g++.} contain @sc{g++} tests.  To add a new test, just put it in
any directory (including, if you wish, an entirely new directory) whose
name follows this convention.

A test is any file in an appropriately named subdirectory whose name
ends in @samp{.exp} (the conventional way of naming @code{expect}
scripts).  These simple naming conventions make it as simple as possible
to install new tests: all you must do is put the test in the right
directory.

@code{runtest} sorts the tests in each subdirectory by name (using the
Tcl @code{lsort} command) and runs them in the resulting order.

@node Builtins
@section Built in functions
@cindex built in functions

@code{runtest} provides these Tcl functions for use in test scripts.
You can also use any standard @code{expect} or Tcl function.

@ftable @code
@item setup_xfail "@var{config} @r{[}@var{bugno}@r{]}"
@cindex declare expected failure
@cindex failure, expected
Declares that the test is expected to fail on a particular set of
configurations.  The @var{config} argument must be a list of full
three-part @code{configure} target name; in particular, you may not use
the shorter nicknames supported by @code{configure} (but you can use the
common shell wildcard characters to specify sets of names).  The
@var{bugno} argument is optional, and used only in the logging file
output; use it as a link to a bug-tracking system such as @sc{prms}
(@pxref{Overview,, Overview, prms.info, Tracking Bugs With PRMS}).

@quotation
@emph{Warning:} you must call either @code{pass} or @code{fail} after
using @code{setup_xfail} in a test case.  Otherwise, the
expected-failure declaration applies to whatever test runs next,
leading to surprising results.
@end quotation

@item ishost "@var{host}"
@cindex host configuration test
Tests for a particular @emph{host} environment.  If the currently
configured host matches the argument string, the result is @code{1};
otherwise the result is @code{0}.  @var{host} must be a full three-part
@code{configure} host name; in particular, you may not use the shorter
nicknames supported by @code{configure} (but you can use wildcard
characters to specify sets of names).

@item istarget "@var{target}"
@cindex target configuration test
Tests for a particular @emph{target} environment.  If the currently
configured target matches the argument string, the result is @code{1};
otherwise the result is @code{0}.  @var{target} must be a full
three-part @code{configure} target name; in particular, you may not use
the shorter nicknames supported by @code{configure} (but you can use
wildcard characters to specify sets of names).

@item isnative
@cindex native configuration test
Tests whether the current configuration has the same host and target.
When it runs in a @emph{native} configuration this procedure returns a
@code{1}; otherwise it returns a @code{0}.

@item load_lib "@var{library-file}"
@cindex load library file
Loads the file @var{library-file} by searching a path. If DejaGnu
has been installed, it looks in a path starting with the installed
library directory. If DejaGnu is not installed, it defaults to the current
directory. It then looks in the current directory for a directory called
@code{lib}. If there are duplicate definitions, the last one loaded takes
precedence over the earlier ones.

@item pass "@var{string}"
@cindex @code{pass} builtin function
Declares a test to have passed.  @code{pass} writes in the 
log files a message beginning with @samp{PASS} (or @code{XPASS}, if
failure was expected), appending the argument @var{string}.

@item fail "@var{string}"
@cindex @code{fail} builtin function
Declares a test to have failed.  @code{fail} writes in the 
log files a message beginning with @samp{FAIL} (or @code{XFAIL}, if
failure was expected), appending the argument @var{string}.

@item error "@var{string}"
@cindex @code{error} builtin function
Declares detection of a severe error in the test case itself.
@code{error} writes in the log files a message beginning with
@samp{ERROR}, appending the argument @var{string}.

@item warning "@var{string}"
@cindex @code{warning} builtin function
Declares detection of a minor error in the test case itself.
@code{warning} writes in the log files a message beginning with
@samp{WARNING}, appending the argument @var{string}.  Use @code{warning}
rather than @code{error} for cases (such as communication failure
to be followed by a retry) where the test case can recover from the
error.

@ignore
@c FIXME?  Ignore for now because it seems this is meant for use by
@c runtest, not scripts.  Perhaps move into eventual section on runtest
@c internals?  Note may then want to talk about "sum", which prints summary
@c *and* cleans up w/alldone.
@item alldone
@cindex quitting DejaGnu from a test case
@cindex @code{alldone} builtin function
Exit from DejaGnu.  @code{alldone} cleans up @code{runtest} temporary
files, closes all @code{runtest} file descriptors, and sets the shell
variable @code{status}. 
@c exit codes described in Invoking node; xref or repeat if this used.
@end ignore
@end ftable

@node Target dependent
@section Target dependent functions
@cindex target dependent functions

@c FIXME? These may be renamed to just "start", "load", "exit", and
@c "version" eventually.

Each combination of target and tool requires some target-dependent
procedures.  The names of these procedures have a common form: the tool
name, followed by an underbar @samp{_}, and finally a suffix describing
the procedure's purpose.  For example, a procedure to extract the
version from @sc{gdb} is called @samp{gdb_version}.  @xref{Init module,,
Initialization Module}, for a discussion of how DejaGnu arranges to find
the right procedures for each target.

@ftable @code
@item @var{tool}_start
Starts @var{tool}.  For an interactive tool,
@code{@var{tool}_start} starts and initializes the tool, leaving the
tool up and running for the test cases; an example is @code{gdb_start},
the start function for @sc{gdb}.  For a batch oriented tool, @code{@var{tool}_start} runs the tool, leaving the output in
a variable called @code{comp_output}.  Test scripts can then analyze
@samp{$comp_output} to determine the test results.  An example of this
second kind of start function is @code{gcc_start}, the start function
for @sc{gcc}.

The initialization module @file{@var{tool}_init.exp} usually calls
@code{@var{tool}_start} for interactive tools; for batch-oriented tools,
each individual test script calls @code{@var{tool}_start}.

@item @var{tool}_load
Loads something into a tool. For an interactive tool, this conditions
the tool for a particular test case; for example, @code{gdb_load} loads
a new executable file into the debugger. For batch oriented tools,
@code{@var{tool}_load} may do nothing---though, for example, the
@sc{gcc} support uses @code{gcc_load} to load a binary on the target and
run it.  A variable called @samp{$exec_output} holds the output of any
program run. Writing @code{@var{tool}_load} can be the most complex part
of extending DejaGnu to a new tool or a new target, if it requires much
communication coding or file downloading.

Test scripts call @code{@var{tool}_load}.

@item @var{tool}_exit
Cleans up before @code{runtest} exits. For interactive tools, this
usually ends the interactive session.  You can also use
@code{@var{tool}_exit} to remove any temporary files left over from the
tests.

@code{runtest} calls @code{@var{tool}_exit}.

@item @var{tool}_version
Prints the version label and number for @var{tool}.  This is called by the
procedure that prints the final summary report.  The output should consist
of the full path name of the tool being tested, and its version number.

@code{runtest} calls @code{@var{tool}_version}.
@end ftable

The usual convention for return codes from any of these procedures
(although it is not required by @code{runtest}) is to return @code{0} if
the procedure succeeded, @code{1} if it failed, and @code{-1} if there
was a communication error.

@node Input files
@section The files DejaGnu reads
@cindex input files

There are a few files used by DejaGnu for input. The most important one
is @file{site.exp}, which contains the configuration data.
@file{runtest.exp} implements the main test framework; the
@code{runtest} program used to invoke DejaGnu is a short shell script
generated by @code{make} during the configuration process.

@cindex libraries
The only other general-purpose input files are collections of
@code{expect} utilities kept in the @file{lib} subdirectory.  These
files are not used by all tests, but are used (via the Tcl command
@code{source}) often enough to warrant a common definition.

@c FIXME!! ADD descriptions of tool-specific procs in this section someplace.

@noindent
@kindex remote.exp
@kindex lib/remote.exp
@file{lib/remote.exp} defines these functions, for establishing and
managing communications:

@emph{Procedures to establish a connection:} Each of these procedures
tries to establish the connection up to three times before returning.
Warnings (if retries will continue) or errors (if the attempt is
abandoned) report on communication failures.  The result for any of
these procedures is either @code{-1}, when the connection cannot be
established, or the spawn ID returned by the @code{expect} command
@code{spawn}.

@ftable @code
@item telnet @var{hostname}
@itemx rlogin @var{hostname}
@itemx rsh @var{hostname}
@emph{IP network procedures.} @var{hostname} refers to the IP address or
name (for example, an entry in @file{/etc/hosts}) for this target.  The
procedure names reflect the Unix utility used to establish a connection.

@item tip @var{port}
@emph{Serial line procedure.} Connect using the Unix utility @code{tip}.
@var{port} must be a name from the @code{tip} configuration file
@file{/etc/remote}.  Often, this is called @samp{hardwire}, or something
like @samp{hardwire-a}. This file holds all the configuration data for
the serial port.

@item kermit @var{port} @var{bps}
@emph{Serial line procedure.}  Connect using the program @code{kermit}.
@var{port} is the device name, e.g. @file{/dev/ttyb}.  @var{bps} is
the line speed to use (in bits per second) for the connection.
@end ftable

@noindent
@emph{Procedures to manage a connection:}

@ftable @code
@item tip_download @var{spawnid} @var{file}
Download @file{@var{file}} to the process @var{spawnid} (the value
returned when the connection was established), using the @code{~put}
command under @code{tip}.  Most often used for single board computers
that require downloading programs in @sc{ascii} S-records.  Returns
@code{1} if an error occurs, @code{0} otherwise.

@item exit_remote_shell @var{spawnid}
Exits a remote process started by any of the connection procedures.
@var{spawnid} is the result of the connection procedure that started the
remote process.
@end ftable

@noindent
@kindex utils.exp
@kindex lib/utils.exp
@file{lib/utils.exp} defines these utility procedures:
@ftable @code
@item getdirs @var{path}
@itemx getdirs @var{path} @var{pattern}
Returns a list of all the directories in @var{path} that match
@var{pattern}.  If you do not specify @var{pattern}, @code{getdirs}
assumes @samp{*}.  You may use the common shell wildcard characters in
@var{pattern}.

@item find @var{path} @var{pattern}
Search for files whose names match @var{pattern} (using shell wildcard
characters for filename expansion).  Search subdirectories recursively,
starting at @var{path}.  The result is the list of files whose names
match; if no files match, the result is empty.  Filenames in the result
include all intervening subdirectory names.

@item which @var{binary}
@cindex path lookup
Searches the execution path for an executable file @var{binary}, like
the the BSD @code{which} utility.  This procedure uses the shell
environment variable @samp{PATH}. It returns @code{0} if the binary is
not in the path, or if there is no @samp{PATH} environment variable. If
@var{binary} is in the path, it returns the full path to @var{binary}.

@item grep @var{filename} @var{regexp}
@item grep @var{filename} @var{regexp} line
Search the file called @var{filename} (a fully specified path) for lines contain a match for
regular expression @var{regexp}.  The result is a list of all the lines that
match.  If no lines match, the result is an empty string.
Specify @code{regexp} using the standard regular expression style used by
the Unix utility program @code{grep}.

Use the optional third argument @samp{line} to start lines in the result
with the line number in @var{filename}.

@item prune @var{list} @var{pattern}
@c FIXME! does this obey the usual Tcl quoting conventions?
Remove elements of the Tcl list @var{list}.  Elements are fields
delimited by spaces.   The result is a copy of @var{list}, without any
elements that match @var{pattern}.
You can use the common shell wildcard characters to specify @var{pattern}.
@end ftable

Most tools also have a library file associated with the tool, containing
procedures found useful by other test developers.  These are especially
helpful when the tests span several subdirectories.

@node Output files
@section The files DejaGnu writes
@cindex output files

@code{runtest} always writes two kinds of output files: summary logs and
detailed logs.  The contents of both of these are determined by your
tests.  

For troubleshooting, a third kind of output file is useful: use
@samp{--debug} to request an output file showing details of what
@code{expect} is doing internally.

@menu
* Summary::		Files that summarize tests
* Detail::		Files that contain complete test results
* Debug::		Logging expect internal actions
@end menu

@node Summary
@subsection Summary Log
@cindex summary log

@code{runtest} always produces a summary output file.  This summary
shows the names of all test files run; for each test file, one line of
output from each @code{pass} command (showing status @samp{PASS} or
@samp{XPASS}) or @code{fail} command (status @samp{FAIL} or
@samp{XFAIL}) (or from @code{error} or @code{warning} commands);
trailing summary statistics that count passing and failing tests
(expected and unexpected); and the full pathname and version number of
the tool tested.  (All four possible outcomes, and all errors, are
always reflected in the summary output file, regardless of whether or
not you specify @samp{--all}.)

Summary logs are saved in a subdirectory called @file{summary}; each
summary log is named @file{@var{tool}.sum}---for example, look for a
summary log for @code{nm} tests in @file{summary/nm.sum}.

@noindent
Here is a short sample summary log:

@smallexample
Test Run By rob on Mon May 25 21:40:57 PDT 1992
                === gdb tests ===
Running ./gdb.t00/echo.exp ...
PASS:   Echo test
Running ./gdb.all/help.exp ...
PASS:   help add-symbol-file
PASS:   help aliases
PASS:   help breakpoint "bre" abbreviation
FAIL:   help run "r" abbreviation
Running ./gdb.t10/crossload.exp ...
PASS:   m68k-elf (elf-big) explicit format; loaded
XFAIL:  mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed C types 
                === gdb Summary ===
# of expected passes 5
# of expected failures 1
# of unexpected successes 0
# of unexpected failures 1
/usr/latest/bin/gdb version 4.6.5 -q
@end smallexample

@node Detail
@subsection Detailed Log
@cindex detailed log

@code{runtest} also saves a detailed log file, showing any output
generated by tests as well as the summary output.  @code{runtest} saves
detailed logs in a subdirectory called @file{detail}; each log file is
named @file{@var{tool}.log}---for example, after you run tests on
@code{nm}, look for the detailed log in @file{detail/nm.log}.

@noindent
Here is a brief example showing a detailed log for @sc{g++} tests:

@example
Test Run By rob on Mon May 25 21:40:43 PDT 1992

                === g++ tests ===

--- Running ./g++.other/t01-1.exp ---
        PASS:   operate delete

--- Running ./g++.other/t01-2.exp ---
        FAIL:   i960 bug EOF
p0000646.C: In function `int  warn_return_1 ()':
p0000646.C:109: warning: control reaches end of non-void function
p0000646.C: In function `int  warn_return_arg (int)':
p0000646.C:117: warning: control reaches end of non-void function
p0000646.C: In function `int  warn_return_sum (int, int)':
p0000646.C:125: warning: control reaches end of non-void function
p0000646.C: In function `struct foo warn_return_foo ()':
p0000646.C:132: warning: control reaches end of non-void function

--- Running ./g++.other/t01-4.exp ---
        FAIL:   abort
900403_04.C:8: zero width for bit-field `foo'
--- Running ./g++.other/t01-3.exp ---
        FAIL:   segment violation
900519_12.C:9: parse error before `;'
900519_12.C:12: Segmentation violation
/usr/latest/bin/gcc: Internal compiler error: program cc1plus got fatal
signal

                === g++ Summary ===

# of expected passes 1
# of expected failures 3
# of unexpected successes 0
# of unexpected failures 0
/usr/ps/bin/g++ version cygnus-2.0.1

@end example

@node Debug
@subsection Logging @code{expect} internal actions
@cindex debug log

With the @samp{--debug} option, you can request a log file showing the
output from @code{expect} itself, running in debugging mode. This file
(@file{dbg.log}, in the directory where you start @code{runtest}) shows
each pattern @code{expect} considers in analyzing test output.

This file reflects each @code{send} command, showing the string sent as
input to the tool under test; and each @code{expect} command, showing
each pattern it compares with the tool output.

The log messages for @code{expect} begin with a message of the form

@smallexample
expect: does @{@var{tool output}@} (spawn_id @var{n}) match pattern
@{@var{expected pattern}@}?
@end smallexample

@noindent
For every unsuccessful match, @code{expect} issues a @samp{no} after
this message; if other patterns are specified for the same
@code{expect} command, they are reflected also, but without the first
part of the message (@samp{expect@dots{}match pattern}).  

When @code{expect} finds a match, the log for the successful match ends
with @samp{yes}, followed by a record of the @code{expect} variables set
to describe a successful match.  Here is an excerpt from the debugging
log for a @sc{gdb} test:

@c FIXME! Why is the second spawn_id shown 0 rather than 6?
@example
send: sent @{break gdbme.c:34\n@} to spawn id 6
expect: does @{@} (spawn_id 6) match pattern @{Breakpoint.*at.* file
 gdbme.c, line 34.*\(gdb\) $@}? no
@{.*\(gdb\) $@}? no
expect: does @{@} (spawn_id 0) match pattern @{<return>@}? no
@{\(y or n\) @}? no
@{buffer_full@}? no
@{virtual@}? no
@{memory@}? no
@{exhausted@}? no
@{Undefined@}? no
@{command@}? no
break gdbme.c:34
Breakpoint 8 at 0x23d8: file gdbme.c, line 34.
(gdb) expect: does @{break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: file
gdbme.c, line 34.\r\n(gdb) @} (spawn_id 6) match pattern
@{Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $@}? yes
expect: set expect_out(0,start) @{18@}
expect: set expect_out(0,end) @{71@}
expect: set expect_out(0,string) @{Breakpoint 8 at 0x23d8: file gdbme.c,
line 34.\r\n(gdb) @}
expect: set expect_out(spawn_id) @{6@}
expect: set expect_out(buffer) @{break gdbme.c:34\r\nBreakpoint 8 at
0x23d8: file gdbme.c, line 34.\r\n(gdb) @}
        PASS:   70      0       breakpoint line number in file
@end example

@noindent
This example exhibits three properties of @code{expect} and DejaGnu that
might be surprising at first glance:

@itemize @bullet
@item
Empty output for the first attempted match.  The first set of attempted
matches shown ran against the output @samp{@{@}}---that is, no output.
@code{expect} begins attempting to match the patterns supplied
immediately; often, the first pass is against incomplete output (or
completely before all output, as in this case).

@item
Interspersed tool output.  The beginning of the log entry for the second
attempted match may be hard to spot: this is because the prompt
@samp{(gdb) } appears on the same line, just before the @samp{expect:}
that marks the beginning of the log entry.  

@item
Default patterns.  Most of the patterns shown are fail-safe patterns provided 
by DejaGnu itself.
@end itemize

@code{runtest} first tries the patterns specified explicitly in each
test case; however, if none of these patterns matches, @code{runtest}
tries a default list of patterns designed to try to catch common errors
or oversights in writing test scripts.  

The default patterns (like the debugging log itself) are primarily
useful while developing test scripts.  The actions for these default
patterns produce messages starting with @samp{ERROR} on the
@code{runtest} standard output, and in the detailed log file.  Here is a
list of these default patterns:

@table @code
@item @samp{@{<return>@}}
Checks for an unexpected @key{RET} prompt.  This may happen, for
example, if the tested tool can filter output through a pager.

@item @samp{@{\(y or n\) @}}
A common pattern in prompts requiring user input.  This will trap some
unexpected prompts from a program.

@item @samp{@{buffer_full@}}
Detects overflow of the internal @code{expect} buffer for output from
the tested tool.  The default buffer size is 2000 characters.  You can
specify a larger buffer, if some of your tests need one, with the
@code{match_max} Tcl function provided by @code{expect}.

@item @samp{@{virtual@} @{memory@} @{exhausted@}}
The message @samp{virtual memory exhausted} from @sc{gdb} (and perhaps
other tested tools) may indicate a problem with the test script itself.
Sometimes a pattern with poor closure (@pxref{Hints,,Hints on Writing a
Test Case}) fills up available memory, leaving very little for the
process that was spawned.

@item @samp{@{Undefined@} @{command@}}
The message @samp{Undefined command} from @sc{gdb} (and perhaps other
tested tools) may be a response to a test script that sends a
nonexistent command.  Sometimes commands disappear as updates are
released; this may also arise from a simple typing mistake in a test
script.
@end table

@node Tests
@chapter How To Write a Test Case
@cindex writing a test case
@cindex test case, writing

@menu
* Writing::			Writing a test case
* Debugging::			Debugging a test case
* Adding::			Adding a test case to DejaGnu
* Hints::			Hints on writing a test case
* Variables::			Special variables used by test cases
@end menu

@node Writing
@section Writing a Test Case

The easiest way to prepare a new test case is to base it on an existing
one for a similar situation.  There are two major categories of tests:
batch or interactive.  Batch oriented tests are usually easier to write.

The @sc{gcc} tests are a good example of batch oriented tests.  All
@sc{gcc} tests consist primarily of a call to a single common procedure,
since all the tests either have no output, or only have a few warning
messages when successfully compiled.  Any non-warning output is a test
failure.  All the C code needed is kept in the test directory.  The test
driver, written in @code{expect}, need only get a listing of all the C
files in the directory, and compile them all using a generic procedure.
This procedure and a few others supporting for these tests are kept in
the library module @file{lib/c-torture.exp}.  Most tests of this kind
use very few @code{expect} features, and are coded almost purely in Tcl.

@noindent
Writing the complete suite of C tests, then, consisted of these steps:

@enumerate
@item 
@cindex Granlund, Torbjorn
@cindex C torture test
Copying all the C code into the test directory.  These tests were based on
the C-torture test created by Torbjorn Granlund (on behalf of the Free
Software Foundation) for @sc{gcc} development.

@item
Writing (and debugging) the shared @code{expect} library code.

@item
Writing the simple test driver: its main task is to search the directory
(using the Tcl procedure @code{glob} for filename expansion with
wildcards) and call a Tcl procedure with each filename.  It also checks
for a few errors from the testing procedure.
@end enumerate

Testing interactive programs is intrinsically more complex.  Tests for most
interactive programs require some trial and error before they are complete.

However, some interactive programs can be tested in a simple fashion
reminiscent of batch tests.  For example, the @sc{gdb} distribution
itself includes a testing procedure that covers most of the DejaGnu
@sc{gdb} tests.  This procedure is very robust, and has already
undergone much more debugging and error checking than many other DejaGnu
test cases.  Accordingly, the best approach was simply to encapsulate
the existing @sc{gdb} tests, for reporting purposes, in DejaGnu tests
that use common library code.

@file{gdb.t10/crossload.exp} is a good example of an interactive test.
@c FIXME!  Check what *kind* of example it is---work-intensive, or generic...

@node Debugging
@section Debugging a Test Case
@cindex debugging a test case
@cindex test case, debugging

@noindent
These are the kinds of debugging information available from DejaGnu:

@enumerate
@item
Output controlled by test scripts themselves, explicitly allowed for by
the test author.  This kind of debugging output appears in the logs
recorded in the @code{detail} directory.  To do the same for new tests,
use the @samp{verbose} variable to control how much output to generate;
this will make it easier for other people running the test to debug it
if necessary.  Whenever possible, if @samp{$verbose} is @code{0}, there
should be no output other than the output from @code{pass}, @code{fail},
@code{error}, and @code{warning}.  Then, to whatever extent is
appropriate for the particular test, allow successively higher values of
@samp{verbose} to generate more information.  Be kind to other
programmers who use your tests: provide for a lot of debugging
information.

@item
Output from the internal debugging functions of Tcl and @code{expect}.
There is a command line options for each; both forms of debugging output
are recorded in the file @code{dbg.log} in the current directory.

Use @samp{--debug} for information from the @code{expect} level; it
generates displays of the @code{expect} attempts to match the tool
output with the patterns specified (@pxref{Debug,,Debug Log}).  This
output can be very helpful while developing test scripts, since it shows
precisely the characters received.  Iterating between the latest attempt
at a new test script and the corresponding @file{dbg.log} can allow you
to create the final patterns by ``cut and paste''.  This is sometimes
the best way to write a test case.

Use @samp{--strace} to see more detail at the Tcl level; this shows how Tcl
procedure definitions expand, as they execute.  The associated number
controls the depth of definitions expanded; see the discussion of
@samp{--strace} in @ref{Invoking runtest,,Running the Tests}.

@item
Finally, if the value of @samp{verbose} is 3 or greater, @code{runtest}
turns on the @code{expect} command @code{log_user}.  This command prints
all @code{expect} actions to the @code{expect} standard output, to the
detailed log file, and (if @samp{--debug} is on) to @file{dbg.log}.
@end enumerate

@node Adding
@section Adding a Test Case to DejaGnu
@cindex adding a test case to DejaGnu

There are two slightly different ways to add a test case. One is to add
the test case to an existing directory. The other is to create a new
directory to hold your test. The existing test directories represent
several styles of testing, all of which are slightly different; examine
the directories for the tool of interest to see which (if any) is most
suitable.

Adding a @sc{gcc} test can be very simple: just add the C code to any
directory beginning with @samp{gcc.} and it runs on the next
@samp{runtest --tool gcc}.

To add a test to @sc{gdb}, first add any source code you will need to
the test directory. Then you can either create a new @code{expect} file,
or add your test to an existing one (any file with a @samp{.exp}
suffix).  Creating a new @samp{.exp} file is probably a better idea if
the test is significantly different from existing tests. Adding it as a
separate file also makes upgrading easier. If the C code has to be
already compiled before the test will run, then you'll have to add it to
the @file{Makefile.in} file for that test directory, then run
@code{configure} and @code{make}.

Adding a test by creating a new directory is very similar:

@enumerate
@item
Create
the new directory. All subdirectory names begin with the name of the
tool to test; e.g. @sc{g++} tests might be in a directory called
@file{g++.other}. There can be multiple test directories that start with
the same tool name (such as @samp{g++}).

@item
Add the new directory name to the @samp{configdirs} definition in the
@file{configure.in} file for the DejaGnu root directory. This way when
make and configure are run, it gets runs on the new directory too.

@item
Add the new test code to the directory. 

@item
To add support in the new directory for configure and make, you must
also create a @code{Makefile.in} and a @code{configure.in}.  @xref{What
Configure Does,,What Configure Does, Cygnus Configure, configure.info}.
@end enumerate

@c FIXME! Expand this sentence to at least a section, maybe a chapter...
The @file{admin} directory contains templates for a few common forms
of test.

@node Hints
@section Hints on Writing a Test Case
@cindex hints on test case writing

There may be useful existing procedures already written for your test in
the @file{lib} directory of the DejaGnu distribution.  @xref{Input files,,The
files DejaGnu reads}.

It is safest to write patterns that match @emph{all} the output
generated by the tested program; this is called @dfn{closure}.  If a
pattern does not match the entire output, any output that remains will
be examined by the @emph{next} @code{expect} command.  In this
situation, the precise boundary that determines which @code{expect}
command sees what is very sensitive to timing between the @code{expect}
task and the task running the tested tool; as a result, the test may
sometimes appear to work, but is likely to have unpredictable results.
The best way to ensure closure is to use the @samp{-re} option for the
@code{expect} command to write the pattern as a full regular
expressions; then you can match the end of output using a @samp{$}. 
It is also a good idea to write patterns that match all available output
by using @samp{.*\} after the text of interest; this will also match any
intervening blank lines.  Sometimes an alternative is to match end of
line using @samp{\r} or @samp{\n}, but this is usually too dependent on
terminal settings.

Always escape punctuation, such as @samp{(} or @samp{"}, in your
patterns; for example, write @samp{\(}.  If you forget to escape
punctuation, you will usually see an error message like @samp{extra
characters after close-quote}.

If you have trouble understanding why a pattern does not match the
program output, try using the @samp{--debug} option to @code{runtest},
and examine the debug log carefully.  @xref{Debug,,Debug Log}.  

Be careful not to neglect output generated by setup rather than by the
interesting parts of a test case.  For example, while testing @sc{gdb},
I issue a send @samp{set height 0\n} command.  The purpose is simply to
make sure @sc{gdb} never calls a paging program.  The @samp{set height}
command in @sc{gdb} does not generate any output; but running @emph{any}
command makes @sc{gdb} issue a new @samp{(gdb) } prompt.  If there were
no @code{expect} command to match this prompt, the output @samp{(gdb) }
begins the text seen by the next @code{expect} command---which might
make @emph{that} pattern fail to match.

To preserve basic sanity, I also recommended that no test ever pass if
there was any kind of problem in the test case.  To take an extreme
case, tests that pass even when the tool will not spawn are misleading.
Ideally, a test in this sort of situation should not fail either.
Instead, print an error message by calling one of the DejaGnu procedures
@code{error} or @code{warning}.

@node Variables
@section Special variables used by test cases
@cindex special variables

Your test cases can use these variables (as well as the variables saved
in @file{site.exp} @pxref{Site specific,,Site configuration}):

@quotation
@emph{These variables are available to all test cases.}
@end quotation

@ftable @code
@item prms_id
The @sc{prms} number identifying a corresponding bugreport.
(@samp{0} if you do not specify it in the test script.)

@item bug_id
An optional bug id; may reflect a bug identification from another
organization.  (@samp{0} if you do not specify it.)

@item subdir
The subdirectory for the current test case.
@end ftable

@quotation
@emph{These variables should never be changed. They appear in most
tests.}
@end quotation

@ftable @code
@item expect_out(buffer)
The output from the last command. This is an internal variable set by
@code{expect}. 

@item exec_output
This is the output from a @code{load} command. This only applies to
tools like @sc{gcc} and @sc{gas} which produce an executable image that
must in turn be executed to complete a test.

@item comp_output
This is the output from a @code{start} command.  This is useful for batch
oriented programs, like @sc{gcc} and @sc{gas}, that may produce interesting
output (warnings, errors) without further interaction.
@end ftable

@node Extending
@chapter Extending DejaGnu

The most common ways to extend the DejaGnu framework are: porting
@code{runtest} to a new host, adding a tool to be tested, and adding
support for testing on a new target.

@menu
* Porting::			Porting DejaGnu
* Adding tools or targets::	How to add a tool or a target
@end menu

@node Porting
@section Porting to a new host
@cindex porting to a new host

The task of porting DejaGnu is basically that of porting Tcl and
@code{expect}. Tcl uses @code{autoconf} so it should port automatically
to most Unix systems. @code{expect} currently uses Cygnus configure,
which requires a makefile stub for each platform.  These makefile
fragments are in the @file{deja-gnu/config} subdirectory of the DejaGnu
source distribution.  By convention, these files have names that begin
with @samp{mt-}; for example, the most general-purpose stub is called
@file{mt-unix}.  If the platform you need is not included, copy the
closest @file{config/mt-*} file and modify it as necessary; then edit
@file{configure.in} to select the new fragment for the appropriate
configuration.

Once Tcl and @code{expect} are ported, DejaGnu should run.  Most system
dependencies are taken care of by using @code{expect} as the main
command shell.

@node Adding tools or targets
@section Adding a tool or a target
@cindex adding a tool
@cindex adding a target

Every tool requires an initialization module for each target testing
environment.  Conversely, supporting a new target requires an
initialization module for each tools you wish to test there.  This
module contains all the operating system dependent code.  Name new
initialization modules following the pattern
@file{@var{target_os}-@var{tool}}.  @xref{Init module,,Initialization
Module}.

Most of the work is in getting the communications right. Communications
code (for several situations involving IP networks or serial lines) is
available in a DejaGnu library file, @file{lib/remote.exp}.
@xref{Input files,,The files DejaGnu reads}.

Usually the best way to write a new initialization module is to edit an
existing initialization module; some trial and error will be required.
If necessary, remember you can use the @samp{--debug} option to see what
is really going on.  

When you code an initialization module, be generous in using print
statements controlled by the level in @samp{$verbose}.

@node Index
@unnumbered Index

@printindex cp

@contents
@bye
