CLHS: Issue EVAL-WHEN-NON-TOP-LEVEL Writeup

#

## Issue EVAL-WHEN-NON-TOP-LEVEL Writeup

    Issue:        EVAL-WHEN-NON-TOP-LEVEL
    Forum:        Compiler
    References:   EVAL-WHEN (CLtL pp69-70),
                  Issue DEFINING-MACROS-NON-TOP-LEVEL
    	      Issue COMPILED-FUNCTION-REQUIREMENTS
    	      Issue IN-PACKAGE-FUNCTIONALITY
    	      Issue LOCALLY-TOP-LEVEL
    Category:     CLARIFICATION/CHANGE
    Edit History: 06-May-88, Version 1 by Sandra Loosemore
                  16-Dec-88, Version 2 by Loosemore (alternate direction)
                  30-Dec-88, Version 3 by Loosemore (minor wording changes)
                  07-Jan-89, Version 4 by Loosemore (update discussion)
                  09-Feb-89, Version 5 by Pitman and Moon (some major changes)
    	      09-Mar-89, Version 6 by Loosemore (clean up wording)
    	      22-Mar-89, Version 7 by Loosemore (order of processing)
    Status:       Ready for release

    Problem Description:

      The current description of how the compiler should handle EVAL-WHEN
      only makes sense when it appears as a top-level form in the file being
      compiled. Is it legitimate for EVAL-WHEN to appear in non-top-level
      locations? Even if it is legitimate, what does it mean?

      Another issue, referred to here as ``the EVAL-WHEN shadowing problem,''
      is that some people have complained that shadowing the symbols EVAL,
      COMPILE, or LOAD means that you have to also either shadow EVAL-WHEN
      and define it to recognize the new symbol, or else you must resign
      yourself to writing (EVAL-WHEN (... LISP:EVAL ...) ...),etc. all over.
      While the goal here is not to solve this problem, it might be possible
      to solve both problems at once.

      There are two proposals presented here, GENERALIZE-EVAL and
      GENERALIZE-EVAL-NEW-KEYWORDS.

    Background/Analysis:

      The proposal which follows was constructed with the following goals
      in mind:

        1. The lexical and dynamic environment for the EVAL-WHEN body should
           be the same for each situation.  That is, the body should ``mean
           the same thing'' regardless of which situation is being processed.

        2. The evaluation context for EVAL-WHEN should be the current
           lexical environment.

        3. At execution time, EVAL-WHEN should always return the result of
           its last form if execution of the body occurred, or NIL if the
           body was not executed.

        4. If a top-level EVAL-WHEN has a LOAD keyword, its body should
           inherit top-level-ness during normal processing. This permits the
           use of (EVAL-WHEN (EVAL COMPILE LOAD) ...) at top-level to mean
           simply "Do whatever would normally be done for this body, but
           also do something at compile time." This, in turn, will later be
           the key to allowing defining forms to be usefully described in
           terms of EVAL-WHEN.

        5. Non-top-level expressions should have no effect until they are
           executed. This is the key to making sure that any necessary
           environment is present. Since the COMPILE keyword forces effects
           to occur earlier than execution time, it follows from this that
           any correct solution must not allow the COMPILE keyword to have
           an effect at other than top-level.

      To accomplish these goals, we formulated the following model:

        The purpose of EVAL-WHEN is to accomodate the fact that some of the
        semantic processing of an expression may usefully be partitioned
        between compile time and run time in some circumstances.

        (EVAL-WHEN (EVAL) <code>)
        describes a general technique for accomplishing some particular goal
        at normal program execution time. However, the pair of expressions
        (EVAL-WHEN (COMPILE) <code-A>)
        (EVAL-WHEN (LOAD) <code-B>)
        can be used to describe an alternate technique for implementing part
        of the effect (A) at compile-time, and part of the effect (B) at
        load-time.

    Proposal (EVAL-WHEN-NON-TOP-LEVEL:GENERALIZE-EVAL):

      Replace the description of EVAL-WHEN with the following:

      EVAL-WHEN ({situation}*) {form}*                      [Special Form]

      The body of an EVAL-WHEN form is processed as an implicit PROGN, but
      only in the situations listed.  Each SITUATION must be a symbol,
      either COMPILE, LOAD, or EVAL.

      The use of COMPILE and LOAD controls whether and when processing
      occurs for top-level forms. The use of EVAL controls whether
      processing occurs for non-top-level forms.

      The EVAL-WHEN construct may be more precisely understood in terms of
      a model of how the file compiler, COMPILE-FILE, processes forms in a
      file to be compiled.

      Successive forms are read from the file by the file compiler using
      READ. These top-level forms are normally processed in what we call
      `not-compile-time' mode. There is one other mode, called
      `compile-time-too' mode, which can come into play for top-level
      forms. The EVAL-WHEN special form is used to annotate a program
      in a way that allows the program doing the processing to select
      the appropriate mode.

      Processing of top-level forms in the file compiler works as follows:

       * If the form is a macro call, it is expanded and the result is
         processed as a top-level form in the same processing mode
         (compile-time-too or not-compile-time).

       * If the form is a PROGN form, each of its body forms is
         sequentially processed as top-level forms in the same processing
         mode.

       * If the form is a COMPILER-LET, MACROLET, or SYMBOL-MACROLET,
         the file compiler makes the appropriate bindings and recursively
         processes the body forms as an implicit top-level PROGN with those
         bindings in effect, in the same processing mode.

       * If the form is an EVAL-WHEN form, it is handled according to
         the following table:

         COMPILE LOAD EVAL compile-time-too Action

           Yes   Yes  --     --             Process body in compile-time-too mode
           No    Yes  Yes    Yes            Process body in compile-time-too mode
           No    Yes  Yes    No             Process body in not-compile-time mode
           No    Yes  No     --             Process body in not-compile-time mode
           Yes   No   --     --             Evaluate body
           No    No   Yes    Yes            Evaluate body
           No    No   Yes    No             do nothing
           No    No   No     --             do nothing

         "Process body" means to process the body as an implicit top-level
         PROGN.  "Evaluate body" means to evaluate the body forms as in
         implicit PROGN in the dynamic execution context of the compiler and
         in the lexical environment in which the EVAL-WHEN appears.

       * Otherwise, the form is a top-level form that is not one of the
         special cases.  If in compile-time-too mode, the compiler first
         evaluates the form and then performs normal compiler processing
         on it.  If in not-compile-time mode, only normal compiler
         processing is performed.  [The nature of this processing is
         defined more precisely in issue COMPILED-FUNCTION-REQUIREMENTS.]
         Any subforms are treated as non-top-level forms.

      Note that top-level forms are guaranteed to be processed in the order
      in which they textually appear in the file, and that each top-level
      form read by the compiler is processed before the next is read.
      However, the order of processing (including, in particular, macro
      expansion) of subforms that are not top-level forms is unspecified.

      For an EVAL-WHEN form that is not a top-level form in the file compiler
      (that is, one of: in the interpreter; in COMPILE; or in the file
      compiler but not at top-level), if the EVAL situation is specified,
      its body is treated as an implicit PROGN.  Otherwise, the EVAL-WHEN
      form returns NIL.

     Clarifications/Consequences:

      The following effects are logical consequences of the above proposal:

       * It is never the case that the execution of a single EVAL-WHEN
         expression will execute the body code more than once.

       * The keyword `EVAL' is a misnomer because execution of
         the body need not be done by EVAL. In compiled code, such as
         (DEFUN FOO () (EVAL-WHEN (EVAL) (PRINT 'FOO)))
         the call to PRINT should be compiled.

       * Macros intended for use in top-level forms should arrange for all
         side-effects to be done by the forms in the macro expansion.
         The macro-expander itself should not do the side-effects.

           Wrong:  (defmacro foo ()
                     (really-foo)
                     `(really-foo))

           Right:  (defmacro foo ()
                     `(eval-when (compile eval load) (really-foo)))

         Adherence to this convention will mean that such macros will behave
         intuitively when placed in non-top-level positions.

       * Placing a variable binding around an EVAL-WHEN reliably captures the
         binding because the `compile-time-too' mode cannot occur (because
         introducing a variable binding would mean we were not at top level).
         For example,

            (LET ((X 3))
              (EVAL-WHEN (EVAL LOAD COMPILE) (PRINT X)))

         will print 3 at execution [load] time, and will not print anything at
         compile time.  This is important so that expansions of DEFUN and
         DEFMACRO can be done in terms of EVAL-WHEN and can correctly capture
         the lexical environment.

            (DEFUN BAR (X) (DEFUN FOO () (+ X 3)))

         might expand into

            (DEFUN BAR (X)
              (PROGN (EVAL-WHEN (COMPILE)
                       (COMPILER::NOTICE-FUNCTION-DEFINITION 'FOO '(X)))
                     (EVAL-WHEN (EVAL LOAD)
                       (SETF (SYMBOL-FUNCTION 'FOO) #'(LAMBDA () (+ X 3))))))

         which would be treated the same as

            (DEFUN BAR (X)
              (SETF (SYMBOL-FUNCTION 'FOO) #'(LAMBDA () (+ X 3))))

         by the above rules.

     Test Cases:

      ;; #1: The EVAL-WHEN in this case is not at top-level, so only the EVAL
      ;;     keyword is considered. At compile time, this has no effect.
      ;;     At load time (if the LET is at top level), or at execution time
      ;;     (if the LET is embedded in some other form which does not execute
      ;;     until later) this sets (SYMBOL-FUNCTION 'FOO1) to a function which
      ;;     returns 1.

      (LET ((X 1))
        (EVAL-WHEN (EVAL LOAD COMPILE)
          (SETF (SYMBOL-FUNCTION 'FOO1) #'(LAMBDA () X))))

      ;; #2: If this expression occurs at the top-level of a file to be compiled,
      ;;     it has BOTH a compile time AND a load-time effect of setting
      ;;     (SYMBOL-FUNCTION 'FOO2) to a function which returns 2.

      (EVAL-WHEN (EVAL LOAD COMPILE)
        (LET ((X 2))
          (EVAL-WHEN (EVAL LOAD COMPILE)
            (SETF (SYMBOL-FUNCTION 'FOO2) #'(LAMBDA () X)))))

      ;; #3: If this expression occurs at the top-level of a file to be compiled,
      ;;     it has BOTH a compile time AND a load-time effect of setting the
      ;;     function cell of FOO3 to a function which returns 3.

      (EVAL-WHEN (EVAL LOAD COMPILE)
        (SETF (SYMBOL-FUNCTION 'FOO3) #'(LAMBDA () 3)))

      ;; #4: This always does nothing. It simply returns NIL.

      (EVAL-WHEN (COMPILE)
        (EVAL-WHEN (COMPILE)
          (PRINT 'FOO4)))

      ;; #5: If this form occurs at top-level of a file to be compiled, FOO5 is
      ;;     printed at compile time. If this form occurs in a non-top-level
      ;;     position, nothing is printed at compile time. Regardless of context,
      ;;     nothing is ever printed at load time or execution time.

      (EVAL-WHEN (COMPILE)
        (EVAL-WHEN (EVAL)
          (PRINT 'FOO5)))

      ;; #6: If this form occurs at top-level of a file to be compiled, FOO6 is
      ;;     printed at compile time.  If this form occurs in a non-top-level
      ;;     position, nothing is printed at compile time. Regardless of context,
      ;;     nothing is ever printed at load time or execution time.

      (EVAL-WHEN (EVAL LOAD)
        (EVAL-WHEN (COMPILE)
          (PRINT 'FOO6)))

     Rationale:

      This is compatible with any guarantees made by CLtL, and extends the
      behavior usefully to non-top-level situations.

      This gives a useful meaning to EVAL-WHEN that supports useful and
      predictable behavior if defining macros are used in a non-top-level
      situation.

      The constraints on the order in which top-level forms are processed
      ensure that the compile-time effects of defining macros and EVAL-WHENs
      at the beginning of the file are visible during the processing of
      forms that appear later in the file, which is what most users expect.
      Leaving the order of processing of non-top-level forms unspecified
      allows the compiler to perform certain kinds of transformations that
      change the textual order of subforms.  Users should not depend on
      side-effects from macros that require them to be expanded in any
      particular order.

    Proposal (EVAL-WHEN-NON-TOP-LEVEL:GENERALIZE-EVAL-NEW-KEYWORDS):

      As in GENERALIZE-EVAL, but rename the EVAL keyword to :EXECUTE,
      the COMPILE keyword to :COMPILE-TOPLEVEL, and LOAD keyword to
      :LOAD-TOPLEVEL.

      Deprecate the use of keywords EVAL, COMPILE, and LOAD to EVAL-WHEN.
      For compatibility, they are supported in EVAL-WHEN
      at top-level, but their meaning is not defined elsewhere.

     Rationale:

      The fact that the situation keywords chosen are not the same as
      those now used means that the change can be added in a way that
      is truly upward compatible (not only with CLtL but with existing
      practice in implementations which have chosen to extend or `clarify'
      the definition given in CLtL) since the meaning of EVAL, COMPILE,
      and LOAD in non-top-level situations (which was never spelled
      out in CLtL) can legitimately differ from the meaning of these
      new keywords.

      Using other names and/or the keyword package for the names of
      situations solves the EVAL-WHEN shadowing problem.

      The name `execute' does not promote the confusion that the body of an
      EVAL-WHEN must be executed only in the evaluator. It also does not
      promote the confusion that the body of an EVAL-WHEN, regardless of when
      executed, must run interpreted.

      The names `compile-toplevel' and `load-toplevel' emphasize the fact
      that these cases are not interesting in non-top-level positions.

    Current Practice:

      In Symbolics Genera, the interpreter permits EVAL-WHEN in non-top-level
      positions in a way that is compatible with this proposal but both the
      COMPILE and COMPILE-FILE functions complain about EVAL-WHEN in a
      non-top-level position.

      Both Lucid Common Lisp and Kyoto Common Lisp already interpret the
      EVAL keyword to mean "execute" in non-top-level situations.  Both of
      these implementations also make (EVAL-WHEN (LOAD) ...) suppress
      compile-time "magic" from defining macros such as DEFMACRO.

      IIM describes its EVAL-WHEN as:
       (defmacro eval-when (situations &body body &environment env)
         (if (not (compiler-environment-p env))
             (when (member 'eval situations) `(progn ,@body))
             (progn
               (when (member 'compile situations)
                 (if (compiler-at-top-level-p env)
                     (mapc #'eval body)
                     (warn "Top-level form encountered at non-top-level.")))
               (when (member 'load situations) `(progn ,@body)))))
      Note that the interpretation of the EVAL situation and the nesting
      behavior is different.

    Cost to Implementors:

      The actual change to EVAL-WHEN in both cases is probably fairly
      localized and straightforward to make in most or all implementations.

      The second-order costs of proposal GENERALIZE-EVAL will vary depending
      on whether existing implementations have extended the definition of
      EVAL-WHEN in incompatible ways. If an implementation has made such
      extensions, there may be user and system code which depends on them
      and the cost of converting that code may be non-trivial. There is
      presumably also documentation impact.

      Proposal GENERALIZE-EVAL-NEW-KEYWORDS avoids most or all of the
      second-order costs of proposal GENERALIZE-EVAL.

      The compiler processing for top-level forms might be implemented
      something like:

      ;;; Forms read by the file compiler are passed to PROCESS-TOP-LEVEL-FORM
      ;;;    with a env compile-time-too both NIL.

      (defun process-top-level-form (form env compile-time-too)
          (setq form (macroexpand form env))
          (cond ((not (consp form))
                 nil)
                ((eq (car form) 'progn)
                 (dolist (f (cdr form))
                     (process-top-level-form f env compile-time-too)))
                ((eq (car form) 'compiler-let)
                 (process-compiler-let form env compile-time-too))
                ((eq (car form) 'macrolet)
                 (process-macrolet form env compile-time-too))
                ((eq (car form) 'symbol-macrolet)
                 (process-symbol-macrolet form env compile-time-too))
                ((eq (car form) 'eval-when)
                 (process-eval-when form env compile-time-too))
                (t
                 (if compile-time-too
                     (internal-eval form env))
                 (compile-form form env))
                ))

      (defun process-eval-when (form env compile-time-too)
          (let* ((situations  (cadr form))
                 (body        (cddr form))
                 (compile-p   (member 'compile situations))
                 (load-p      (member 'load situations))
                 (eval-p      (member 'eval situations)))
              (cond ((or (and compile-p load-p)
                         (and eval-p load-p compile-time-too))
                     (process-top-level-form `(progn ,@body) env t))
                    (load-p
                     (process-top-level-form `(progn ,@body) env nil))
                    ((or compile-p
                         (and eval-p compile-time-too))
                     (dolist (f body)
                         (internal-eval f env)))
                    (t
                     nil))))

      ;;; PROCESS-COMPILER-LET, PROCESS-MACROLET, and PROCESS-SYMBOL-MACROLET
      ;;;    do the obvious things.
      ;;; INTERNAL-EVAL evaluates "form" in lexical environment "env".

    Cost to Users:

      Technically, none. Either proposal is technically upward compatible
      with CLtL.

      Proposal GENERALIZE-EVAL might force some extended implementations to
      change incompatibly. As such, some users who depend on
      implementation-dependent extensions might have to adjust their code
      somewhat to deal with those changes.

      Proposal GENERALIZE-EVAL-NEW-KEYWORDS does not force implementations
      to change incompatibly, so has no forced impact on users.

    Cost of Non-Adoption:

      EVAL-WHEN is a mess. Using it as the low-level substrate into which
      defining macros should expand, and guaranteeing any predictable effects
      of those macros in non-top-level situations is currently difficult and
      would continue to be so in the absence of some resolution on this issue.

    Benefits:

      The costs of non-adoption would be avoided:  it would be possible to
      use EVAL-WHEN in many situations where it cannot currently be used
      reliably.

      The portability of many existing tools which use EVAL-WHEN internally
      in macros will be enhanced.

    Aesthetics:

      This generalization of the meaning makes the purpose and uses of
      EVAL-WHEN less mysterious. In that sense, aesthetics are simplified
      somewhat.

    Discussion:

      The cleanup issue LOCALLY-TOP-LEVEL would make LOCALLY also "pass
      through" top-level-ness to its body.  The reason why that is not
      addressed in this issue is that it involves making LOCALLY a special
      form.

      Pitman and Moon don't care whether we say `top level,' `top-level,' or
      `toplevel.' The spelling choices in this writeup are arbitrary. If
      necessary, the proposal GENERALIZE-EVAL-NEW-KEYWORDS could be amended
      to propose :COMPILE-TOP-LEVEL, etc.

      Pitman, Moon, and Bob Laddaga (a Symbolics Cloe implementor) support
      both of these proposals.  Pitman and Laddaga have a preference for
      GENERALIZE-EVAL-NEW-KEYWORDS.  Moon is neutral about which should be
      preferred.

      Sandra Loosemore says:
        I still feel somewhat uncomfortable with the definition of EVAL-WHEN
        presented here, mostly because its nesting behavior is so unintuitive
        (as in test case number 6).  We have also had a hard time in deciding
        what the term "top-level" really means; the definition presented here
        is rather arbitrary.  However, since we have run out of time in which
        to come up with acceptable alternatives, I'm willing to go along with
        proposal GENERALIZE-EVAL.  It is compatible with the description in
        CLtL but presented in a more coherent way, and I think it is an
        improvement.  On the other hand, I don't really like the idea of
        changing the names of the keywords; if we are going to make an
        incompatible change, the right thing to do would be to throw out
        EVAL-WHEN entirely and start from scratch.

      Treating MACROLET and SYMBOL-MACROLET (and possibly LOCALLY)
      complicates the treatment of top-level DEFMACROs and other
      defining macros that cause functions to be created at compile-time
      (because the lexical environment the functions are defined in may
      not be null).  See issue DEFINING-MACROS-NON-TOP-LEVEL for details.-------

Copyright 1996-2005, LispWorks Ltd. All rights reserved.