  Reference Manual for TESS Version 0.3.0
  Michael S. Noble, mnoble@space.mit.edu
  Feb 14, 2007
  ____________________________________________________________

  Table of Contents

   Preface
   Introduction

  1. Motivation
     1.1 Model: Multiple Tests per Script
     1.2 Model: Single Test per Script
     1.3 Consistency and Reuse
     1.3 Examples

  2. Scripts
  3. Output
  4. Analysis
     4.1 tess-common.sl
     4.2 Type Mismatch Exceptions
     4.3 Stack Maintenance
     4.4 Results Summary
     4.5 Exit Status
     4.5 Function Reference

  5. tess_auto_summarize
  6. tess_add_eval_paths
  7. tess_add_import_paths
  8. tess_catch_type_errors
  9. tess_invoke
  10. tess_load_component
  11. tess_summary
  11. Utility Script Reference
  12. tessrun


  ______________________________________________________________________



  [1m1.  Preface[0m



         TESS is the (Te)st (S)ystem for (S)-Lang.

         Copyright (C) 2004-2007  Massachusetts Institute of Technology
         Michael S. Noble <mnoble@space.mit.edu>

         This software was partially developed at the MIT Center for Space
         Research, under contract SV1-61010 from the Smithsonian Institution.

         Permission to use, copy, modify, distribute, and sell this software
         and its documentation for any purpose is hereby granted without fee,
         provided that the above copyright notice appear in all copies and
         that both that copyright notice and this permission notice appear in
         the supporting documentation, and that the name of the Massachusetts
         Institute of Technology not be used in advertising or publicity
         pertaining to distribution of the software without specific, written
         prior permission.  The Massachusetts Institute of Technology makes
         no representations about the suitability of this software for any
         purpose.  It is provided "as is" without express or implied warranty.

         THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY DISCLAIMS ALL WARRANTIES
         WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF
         MERCHANTABILITY AND FITNESS.  IN NO EVENT SHALL THE MASSACHUSETTS
         INSTITUTE OF TECHNOLOGY BE LIABLE FOR ANY SPECIAL, INDIRECT OR
         CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
         OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
         NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
         WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.



  [1m2.  Introduction[0m


  TESS is the (Te)st (S)ystem for (S)-Lang, which aims at reducing the
  workload and [4mad-hoc[24m nature of regression testing S-Lang software (
  <http://www.s-lang.org>), by collecting common testing elements into a
  single, easy-to-use framework.  TESS provides the S-Lang developer
  nominal mechanisms for tailoring the S-Lang environment and invoking
  functions with arbitrary inputs, while transparently inspecting and
  cleaning the stack, gathering pass/fail statistics, and providing
  error recovery from exceptions.

  TESS is primarily a development tool, so we assume the reader is
  familiar with scripting in S-Lang.  Knowledge of how to create S-Lang
  modules is also helpful.  Since TESS is intended to assist in
  automating and simplifying regression testing, it is most often
  utilized in conjunction with the make tool via a Makefile.  If these
  are unfamiliar terms then you will benefit from learning about them
  first prior to reading further.

  Please note that as of version 0.3.0 TESS no longer supports S-Lang 1.
  This enables TESS to be implemented as purely interpreted code (i.e.
  S-Lang and shell scripts, no binaries), and makes it smaller, easier
  to build & use, as well as easier to bundle in other packages (only 2
  small text files are required).


  [1m2.1.  Motivation[0m


  Suppose you had a function defined within some file add.sl as


               define add()
               {
                  variable i, j;
                  if (_NARGS < 2) usage("add(i,j)");
                  (i,j) = ();
                  return i+j;
               }



  and that you wanted to exercise it in automated fashion on a range of
  input conditions.  To test it with insufficient arguments, for
  instance, one might write a script add.t


               () = evalfile("./add");
               add(1);



  which, when invoked from slsh as


               slsh ./add.t



  yields something like


          Usage: add(i,j)
          called from line 2, file: ./add.t



  [1m2.1.1.  Model: Multiple Tests per Script[0m


  Ok, so far things look reasonable.  Now, suppose you appended the line


               add(1,"two");



  to the test and reran it in slsh.  Curiously enough, the output gener-
  ated would look no different from above.  Why?  Because the first
  add() call generates a usage exception, which causes the interpreter
  to unwind the S-Lang stack and then exit.  In other words, the second
  add() call [4mis[24m [4mnever[24m [4meven[24m [4mexecuted![24m One way to address this is to mod-
  ify the script by adding an ERROR_BLOCK, such as:


               () = evalfile("./add");

               define test1()
               {
                  ERROR_BLOCK { _clear_error(); return; }
                  add(1);
               }

               test1;
               add(1, "two");



  Now both tests will be exercised when the script is invoked, generat-
  ing a usage exception in the first case and a type mismatch error in
  the second.  Progress, for sure, but the script has grown longer, and
  we needed to introduce a wrapper for the first test case in order to
  use the error block.  In this model, where one script contains multi-
  ple test cases, each test would need to be invoked within such a wrap-
  per, which explains why TESS offers the tess_invoke() function.


  [1m2.1.2.  Model: Single Test per Script[0m


  An alternative to the model used above would be to each test case
  within its own .t file and invoke slsh upon each.  This approach
  avoids the need for error blocks, but introduces a number of other
  concerns which collectively steer the author towards a preference for
  the first model.

  For example, the resultant file proliferation makes it more cumbersome
  to enumerate/name and effectively organize the test suite.  Packages
  of only moderate size might conceivably contain scores or perhaps
  hundreds of relatively tiny files, needlessly wasting disk resources,
  increasing sizes of software distributions, and wasting CPU cycles by
  launching slsh [4monce[24m [4mper[24m [4mtest[24m instead of only [4monce[24m [4mper[24m [4mgroup[24m [4mof[24m [4mtests[24m.
  Moreover, each test invocation would also need to load both TESS and
  the package being tested, resulting in yet more CPU waste and code
  duplication.

  Identifying semantically related test cases would not be as easy,
  since groupings are now distinguished by like-named files within a
  directory, instead of by cohabitation of test cases within a single
  file.

  Collecting useful failure statistics becomes more difficult, since in
  this model aggregate counts can be be obtained only by metascripts,
  e.g. invoked within Makefiles at the operating system level to keep
  track of the 1 or 0 returned from each test, instead of within the
  test scripts themselves.  In contrast, the tess_invoke function
  mentioned above transparently tallies pass/fail statistics.

  For small test suites these issues may be negligible, but as packages
  grow their cumulative effect may not be so easily ignored, making it
  better to cultivate the preferable habit of "starting clean," rather
  than one of "cleaning up later".


  [1m2.1.3.  Consistency and Reuse[0m


  TESS emerged as the coalescence of scripts used in testing a number of
  existing S-Lang packages. In fact, development versions of packages
  such as SLgtk and SLxpa have already been migrated from their original
  testing scheme towards TESS, and as such they serve as the wealthiest
  source of supplemental examples to this documentation.

  By distilling common patterns from existing test suites into a self-
  contained distribution TESS fosters reuse, reduces duplicative busy
  work, and hopefully serves to ease the burden of testing (perhaps one
  of the least loved aspects of writing and maintaining software).



  [1m3.  Examples[0m


  Before continuing it is worthwhile noting that the examples in this
  chapter are bundled within the examples directory of the TESS
  distribution (along with a Makefile and baseline [4mtest.ref[24m output), and
  that they have been written under the assumption that an operational
  version of TESS is installed on your system.

  It should also be noted that a TESS script is merely a S-Lang script
  which has, somewhere along its execution path, loaded TESS.  Normally
  S-Lang scripts are named with a .sl suffix, however the author has
  adopted the convention of suffixing them with .t instead.  Not only
  does this serve as a useful mnemonic, it also prevents tests from
  being accidently loaded by S-Lang through its default script loading
  mechanism (fostering the use of nearly identical names for both test
  scripts and the package components which they exercise).


  [1m3.1.  Scripts[0m

  Let's begin our example by augmenting the add() function defined above
  with a corresponding subtract() function, and migrating the definition
  of both into a S-Lang script named sillymath.sl, as follows:


               define add()
               {
                  variable i, j;
                  if (_NARGS < 2) usage("add(i,j)");
                  (i,j) = ();
                  return i+j;
               }

               define subtract()
               {
                  variable i, j;
                  if (_NARGS < 2) usage("subtract(i,j)");
                  (i,j) = ();
                  return i - j;
               }



  Two TESS scripts which more thoroughly exercise sillymath.sl, then,
  are:


       add.t:

           require("tess");

           tess_invoke(1, &add);
           tess_invoke(1, &add, "hi there!");
           tess_invoke(1, &add, 2);
           tess_invoke(0, &add, 2, 3);

           tess_invoke(1, &add, "one", 2);
           tess_invoke(0, &add, "hello", " there!");



  subtract.t:

      require("tess");

      variable f = &subtract;
      tess_invoke(1, f);
      tess_invoke(1, f, "hi there!");
      tess_invoke(1, f, 2);
      tess_invoke(0, f, 2, 3);

      tess_invoke(1, f, "one", 2);
      tess_invoke(1, f, "hello", " there!");



  [1m3.2.  Output[0m

  The first of these should emit feedback resembling that given below.
  Since the result of add.t closely resembles that of subtract.t
  (differing only in that case 5 signals an exception in one but not the
  other, since string subtraction is undefined while string addition is
  equivalent to concatenation) we omit output from the latter.


           Usage: add(i,j)

           Test Case 1: add: PASSED (SHOULD signal error, DID)

           Usage: add(i,j)

           Test Case 2: add: PASSED (SHOULD signal error, DID)

           Stack Contents:
           (0)[String_Type]:hi there!

           Usage: add(i,j)

           Test Case 3: add: PASSED (SHOULD signal error, DID)

           Stack Contents:
           (0)[Integer_Type]:2


           Test Case 4: add: PASSED (SHOULD NOT signal error, DID NOT)

           Stack Contents:
           (0)[Integer_Type]:5

           S-Lang Error: Type Mismatch: String_Type + Integer_Type is not possible

           Test Case 5: add: PASSED (SHOULD signal error, DID)


           Test Case 6: add: PASSED (SHOULD NOT signal error, DID NOT)

           Stack Contents:
           (0)[String_Type]:hello there!


           =============== add Test Summary ===============

                       Number of Failures: 0
                       Number of Passes  : 6


  [1m3.3.  Analysis[0m

  In the above tests the absence of an explicit ERROR_BLOCK, and the
  cleaner code which results, should be immediately apparent.   For
  example, add.t contains one fewer non-blank lines than does the
  example in section ``Motivation'', while tripling the number of cases
  tested (6 versus 2).  Also revealed is the fact that tess_invoke() is
  the single most important function in the interface.


  [1m3.3.1.  tess-common.sl[0m

  A more subtle point of interest is that sillymath.sl does not appear
  to be loaded by either test script.   How, then, do they function?
  The answer is that the evalfile() has been pushed into a file tess-
  common.sl, which TESS will transparently load if found within the
  current directory at startup.

  This exploits a common pattern within test suites, namely that prior
  to testing [4many[24m component within a package a test script must first
  load the package itself.  Furthermore, tess-common.sl can be used to
  define variables, data structures, or functions that might be commonly
  used amongst all test scripts within a given suite.


  [1m3.3.2.  Type Mismatch Exceptions[0m

  Consider the case


               tess_invoke(1, &add, "one", 2);



  defined for add() (and the similar test defined for subtract()), which
  attempts to add an Integer_Type to a String_Type and results in a type
  mismatch exception.  Even with an ERROR_BLOCK defined S-Lang 1.x would
  not, on its own, catch this exception (although S-Lang 2.x will).
  This is because S-Lang 1 regards type mismatches as fatal errors, so
  the interpreter will not permit execution to continue after they
  occur.

  Because it can be useful to test such conditions, however, TESS
  relaxes this constraint by installing an error handler which resets S-
  Lang 1 internal state and allows scripts to continue processing.
  Older versions of TESS installs this handler by default at startup,
  but as noted in the function reference it may be deactivated (or
  reactivated) by calling tess_catch_type_errors().


  [1m3.3.3.  Stack Maintenance[0m


  After each test TESS also reports the number and type (or value, for
  objects which are not aggregates) of any items remaining on the S-Lang
  stack.  Consider the line


               Stack Contents:
               (0)[String_Type]:hello there!



  present in the output of add.t above.  It shows that after test case 6
  has completed the S-Lang stack contains 1 item (as it should), namely
  the result of concatenating the strings "hello" and " there!".

  This provides an excellent way of validating the return values of
  exercised functions, without requiring any additional work on the part
  of the test developer.  Conversely, this feature also identifies
  functions which create unintended side effects by leaving detritus on
  the stack.


  [1m3.3.4.  Results Summary[0m

  Another point of interest from the add.t output is that the pass/fail
  statistics of each test script are automatically summarized, again
  with zero work required on the part of the test developer.  This
  happens because TESS installs an [4mexit[24m [4mhandler[24m which, by default,
  transparently calls tess_summary when the test application terminates.
  As noted in the function reference, this behavior may be customized.


  [1m3.3.5.  Exit Status[0m


  If the test application offers an exit intrinsic then TESS will invoke
  it at completion time, passing in the number of failures observed
  while running the script.  This enables higher-level Makefiles, or
  tessrun, to take appropriate action, such as terminating when a non-
  zero status is returned.



  [1m4.  Function Reference[0m



  [1m4.1.  tess_auto_summarize[0m


      [1mSynopsis[0m
        Turn automatic test suite summarization on or off

      [1mUsage[0m
        tess_add_eval_paths( [0 | 1.])

      [1mDescription[0m

      [1mNotes[0m

      [1mSee Also[0m
        tess_summary


  [1m4.2.  tess_add_eval_paths[0m


      [1mSynopsis[0m
        Add one or more directories to the S-Lang evalfile() search path

      [1mUsage[0m
        tess_add_eval_paths( path1, [path2, ...])

      [1mDescription[0m
        This function is a convenience wrapper around the
        set_slang_load_path() function, making it cleaner and simpler to
        augment the list of directories searched by the S-Lang
        interpreter when evalfile() is invoked with an ambiguous file
        specification.

      [1mNotes[0m
        TESS automatically appends the current working directory, as
        well as ../src, ../share, and ../packages to the load path.

      [1mSee Also[0m
        tess_add_import_paths


  [1m4.3.  tess_add_import_paths[0m


      [1mSynopsis[0m
        Add one or more directories to the S-Lang import() search path

      [1mUsage[0m
        tess_add_import_paths( path1, [path2, ...])

      [1mDescription[0m
        This function is a convenience wrapper around the
        set_import_module_path() function, making it cleaner and simpler
        to augment the list of directories searched by the S-Lang
        interpreter when import() is invoked.

      [1mNotes[0m
        TESS automatically appends ../src to the import path.

      [1mSee Also[0m
        tess_add_eval_paths

  [1m4.4.  tess_catch_type_errors[0m


      [1mSynopsis[0m
        Give S-Lang ERROR block mechanism the ability to catch type
        mismatch errors

      [1mUsage[0m
        tess_catch_type_errors( [yes_or_no] )

      [1mDescription[0m
        This function augments the S-Lang ERROR block mechanism, giving
        it the ability to catch type mismatch exceptions (which S-Lang
        1.x formally considers uncatchably fatal).  This feature is
        useful for a test framework, since it allows functions to be
        safely exercised against a wide variety of types.

        If the first passed argument evaluates to a boolean TRUE then
        the function will enable type error catching.  If either zero
        arguments are passed, or the first argument evaluates to boolean
        FALSE, then type error catching will be disabled.

      [1mNotes[0m
        This function is deprecated, as in S-Lang 2 all exceptions may
        be caught.

      [1mSee Also[0m
        tess_invoke


  [1m4.5.  tess_invoke[0m


      [1mSynopsis[0m
        Execute a test case

      [1mUsage[0m
        tess_invoke( expected_to_fail, function_ref [, arg1, arg2, ...])

      [1mDescription[0m
        Invoke the given function (by dereference), optionally passing
        in one or more arguments.  The first parameter, whose value
        should be either zero or one, indicates whether the function is
        expected to signal an error when invoked in the manner given.

        If the actual result of the call matches the expected result
        then the test case is said to "pass," otherwise it is said to
        "fail".  It is important to understand this: a failed test case
        is not indicated by an error signal itself, but rather by
        whether or not the test case expected an error to be signaled.

      [1mNotes[0m

      [1mSee Also[0m
        tess_catch_type_errors, tess_summary


  [1m4.6.  tess_load_component[0m


      [1mSynopsis[0m
        Evaluate the named S-Lang script, and set the test component
        name accordingly

      [1mUsage[0m
        tess_load_component(filename)
      [1mDescription[0m
        This function attempts to evalfile() the named script, using the
        usual S-Lang load mechansism, and will set the TESS test
        component name to the filename if found.

        The test component name is printed in the heading of results
        summaries, and uniquely identifies a given test script.
        Typically the test component name is set to the "basename" of
        the test script itself (e.g. a script add.t sets Component =
        "add").  This function provides a means of customizing that
        default behavior while loading additional functionality to be
        exercised within the test script.

      [1mNotes[0m

      [1mSee Also[0m



  [1m4.7.  tess_summary[0m


      [1mSynopsis[0m
        Summarize the results of a suite of tests

      [1mUsage[0m
        Integer_Type tess_summary()

      [1mDescription[0m
        TESS automatically records the pass/fail result of each test
        case executed by tess_invoke.  By default the results of this
        tally are emitted to stdout when tess_summary is called,
        although this may be disabled by calling tess_auto_summarize(0).
        The return value indicates the number of failed tests.

      [1mNotes[0m
        Under normal circumstances it should not be necessary to call
        this function explicitly, since TESS transparently installs an
        exit handler which calls tess_summary at application
        termination.  Its return value is then passed to the operating
        so that, for example, a non-zero status may be used to fatally
        terminate a "make test" goal.


      [1mSee Also[0m
        tess_invoke



  [1m5.  Utility Script Reference[0m



  [1m5.1.  tessrun[0m


  The tessrun script is intended to simplify the invocation of a suite
  of regression tests, and has a command line usage synopsis of:


               tessrun [-v] [application_name]



  Specifying the -v (verbose) flag inhibits the redirection of stderr to
  /dev/null, while specifying an application_name overrides the default
  use of slsh as the test application.

  The chief benefits of tessrun are that it yields simpler Makefiles
  which require fewer modifications.  For example, rather than having
  the [4mmake[24m [4mtest[24m rule within your regression test Makefile explicitly
  loop over all [4m.t[24m files within its directory, the rule can simply
  invoke tessrun, perhaps as:


               test:
                       tessrun $(TESS_APP) > test.out 2>&1
                       diff test.ref test.out



  Since zero tests are explicitly referenced by name the Makefile need
  not change as new tests are added to the regression suite.  Nor need
  need it be modified simply to execute the test suite within other
  applications (again, the default test application is slsh), since that
  can be achieved here simply by overriding the definition of
  $(TEST_APP) during the invocation of [4mmake[24m:


               unix%  make -e TEST_APP=isis



  The only constraint upon the test application is that it offers the S-
  Lang [4mprovide[24m and [4mrequire[24m package management functions.



                            [1mTable of Contents[0m


  1. Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
  2. Introduction  . . . . . . . . . . . . . . . . . . . . . . . . .   3
  2.1. Motivation  . . . . . . . . . . . . . . . . . . . . . . . . .   3
  2.1.1. Model: Multiple Tests per Script  . . . . . . . . . . . . .   4
  2.1.2. Model: Single Test per Script . . . . . . . . . . . . . . .   4
  2.1.3. Consistency and Reuse . . . . . . . . . . . . . . . . . . .   5
  3. Examples  . . . . . . . . . . . . . . . . . . . . . . . . . . .   6
  3.1. Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . .   6
  3.2. Output  . . . . . . . . . . . . . . . . . . . . . . . . . . .   7
  3.3. Analysis  . . . . . . . . . . . . . . . . . . . . . . . . . .   8
  3.3.1. tess-common.sl  . . . . . . . . . . . . . . . . . . . . . .   8
  3.3.2. Type Mismatch Exceptions  . . . . . . . . . . . . . . . . .   8
  3.3.3. Stack Maintenance . . . . . . . . . . . . . . . . . . . . .   8
  3.3.4. Results Summary . . . . . . . . . . . . . . . . . . . . . .   9
  3.3.5. Exit Status . . . . . . . . . . . . . . . . . . . . . . . .   9
  4. Function Reference  . . . . . . . . . . . . . . . . . . . . . .  10
  4.1. tess_auto_summarize . . . . . . . . . . . . . . . . . . . . .  10
  4.2. tess_add_eval_paths . . . . . . . . . . . . . . . . . . . . .  10
  4.3. tess_add_import_paths . . . . . . . . . . . . . . . . . . . .  10
  4.4. tess_catch_type_errors  . . . . . . . . . . . . . . . . . . .  11
  4.5. tess_invoke . . . . . . . . . . . . . . . . . . . . . . . . .  11
  4.6. tess_load_component . . . . . . . . . . . . . . . . . . . . .  11
  4.7. tess_summary  . . . . . . . . . . . . . . . . . . . . . . . .  12
  5. Utility Script Reference  . . . . . . . . . . . . . . . . . . .  13
  5.1. tessrun . . . . . . . . . . . . . . . . . . . . . . . . . . .  13



