CLHS: Issue DECLARATION-SCOPE Writeup

#

## Issue DECLARATION-SCOPE Writeup

    Proposal DECLARATION-SCOPE:NO-HOISTING was accepted at the January 1989 meeting.

    Issue:         DECLARATION-SCOPE

    References:    Declaration Syntax (CLtL, Section 9.1, pp. 153-157)
    	       LAMBDA-Expressions (CLtL, Section 5.2.2, pp. 59-66)
                   Cleanup issue FLET-DECLARATIONS (accepted)
                   Cleanup issue DECLARE-TYPE-FREE (accepted)

    Category:      CHANGE/CLARIFICATION

    Edit history:  V1: Hornig@Symbolics.COM -- 5 January 1988
                   Version 2, Moon, 2-Feb-1988 (edits based on discussion)
                   Version 3, Moon, 18-Sep-88, for private discussion between JonL and Moon
                   Version 4, JonL, 15-Nov-88 add 2nd proposal; major rewrite.

    Problem description:

    The description of the scope of declarations made with DECLARE is
    unclear (although unambiguous) and arguably a mistake.  At issue is
    whether the scope of some or all of the declarations at the head of
    a special form includes code appearing in any non-body part of that
    special form.  CLtL p.155 attempts to address the issue by classifying
    declarations into two classes -- "pervasive" and "non-pervasive" -- but
    it does not succeed very well.

    A particular question of interest is whether the initial value forms for
    LET, LET*, FLET, LABELS, DO, PROG etc. are included.  The rules of CLtL,
    on some cases, are clear enough to see that a declaration inside the
    special form is "hoisted" up and around the whole form, so as to include
    all the "initial value" forms (and "stepper" forms and "result" forms for
    those constructs that have them).  This means that lexical argument
    variable X in the following function is inaccessible inside the initial
    value forms of the LET:
      (defun bar (x y)	    ;[1] 1st instance of x
        (let ((old-x x)	    ;[2] 2nd instance of x -- same as 1st instance?
    	  (x y))	    ;[3] 3rd instance
          (declare (special x))
          (list old-x x)))
    The declaration intended for the binding of X[3] also alters the
    scoping of the reference of X[2]; likely, this was not an intended
    consequence. [This is a simplification of the example on CLtL p.155].

    In this discussion, the term "body" will include any "stepper" or
    "result" forms, such as might be found in a DO or DO-mumble-SYMBOLS
    construct.  The reasoning for this is that such forms are always
    included in the scope of all name bindings (if any) being established
    by the special form.  They form an extended part of the "body".

    Proposal (DECLARATION-SCOPE:NO-HOISTING)

    Specify that the scope of a declaration located at the head of a special
    form or lambda expression is as follows:
      (1) it always includes the body forms [as well as any "stepper" or
          "result" forms]
      (2) it also includes the scope of the name binding, if any, to which
          it applies [LET, LAMBDA, FLET, DO, etc. introduce "name bindings";
          LOCALLY doesn't.];

    This very straightforward prescription depends on one rather subtle
    point, namely that the scope of name bindings is an already solved
    question.  Whether or not a particular declaration affects an initial
    value form (such as for LET or LET*) depends _solely_ on whether it is
    applied to a variable or function (name) being bound whose scope
    includes such forms.  In this sense, the above specification limits the
    scope of declarations for name bindings to be exactly the scope of the
    name binding itself -- i.e. "like variable".  Thus there would be no
    "hoisting" of the special declarations in the example cited in the
    problem description.  [See the Discussion section for a review of the
    CL rules on variable/function-name scoping in special forms.]

    Those declarations not correlated with any name binding do not cover any
    of the initial-value forms; their scope simply includes the body (as well
    as any "stepper" or "result" forms).  In a sense, the above specification
    limits the scope of these kinds of declarations to be the same as an
    arbitrary name binding in a LET or FLET construct -- i.e. "like variable".
    [See also the issue DECLARE-TYPE-FREE.]

    Thus there is to be no "hoisting" for declarations in special forms or
    lambda expressions; the only initial value forms affected by a declaration
    will be those included indirectly, by the effect, if any, that a
    declaration has on a name binding.

    A question may arise as to what it means for a declaration to "apply to",
    or "be correlated to" a name binding.  As stated above about variable
    scoping, this is an already solved question in Common Lisp; it is not
    the purpose of this proposal to alter, clarify or in any other way bear
    upon the basic _applicability_ rules of declarations in Common Lisp.
    However, a few reminders about these rules will help one understand the
    above specification:
      --  SPECIAL and TYPE declarations never apply to function bindings;
      --  INLINE and FTYPE declarations never apply to variable bindings;
      --  OPTIMIZE never applies to either kind of binding;
      --  (<declaration> X) never applies to a binding of Y.
    The specific rules are for the most part distributed throughout the
    individual declaration definitions.  The name-applicibility issue for
    bindings must be specified independently of how the declaration scoping
    issue is decided, and should not be confused with that scoping issue.

    Proposal (DECLARATION-SCOPE:LIMITED-HOISTING)

    Specify that the scope of a declaration located at the head of a special
    form or lambda expression is as follows:
      (1) it always includes the body forms [as well as any "stepper" or
          "result" forms]
      (2) if the declaration is applicable to a name binding in that form,
          then it is limited to exactly the scope of that name binding [LET,
          LAMBDA, FLET, etc. introduce "name bindings"; LOCALLY doesn't.];
      (3) if the declaration is not applicable to a name binding in that form,
          then it includes all the initial value forms, in addition to the
          body forms.

    This very straightforward prescription depends on one rather subtle
    point, namely that the scope of name bindings is an already solved
    question.  This applies not only to LET and LET* variables, but to the
    &optional, &key and &aux variables of LAMBDA-Expressions.  In this sense,
    the above specification limits the scope of declarations for name bindings
    to be exactly the scope of the name binding itself.  Thus there would be
    no "hoisting" of the special declarations in the example cited in the
    problem description.

    Those declarations not correlated with any name binding act as if they
    were included in a new LOCALLY construct wrapped around the entire
    special form.  Thus they are not scoped like an arbitrary variable
    (or, "name binding") in that special form, but rather are "hoisted" up.

    Whether or not a declaration is "hoisted" up around the special form in
    which it occurs depends on whether or not it is "captured en passant" by
    a correlated name binding.

    A question may arise as to what it means for a declaration to "apply to",
    or "be correlated to" a name binding.  As stated above about variable
    scoping, this is an already solved question in Common Lisp; it is not
    the purpose of this proposal to alter, clarify or in any other way bear
    upon the basic _applicability_ rules of declarations in Common Lisp.
    However, a few reminders about these rules will help one understand the
    above specification:
      --  SPECIAL and TYPE declarations never apply to function bindings;
      --  INLINE and FTYPE declarations never apply to variable bindings;
      --  OPTIMIZE never applies to either kind of binding;
      --  (<declaration> X) never applies to a binding of Y.
    The specific rules are for the most part distributed throughout the
    individual declaration definitions.  The name-applicibility issue for
    bindings must be specified independently of how the declaration scoping
    issue is decided, and should not be confused with that scoping issue.

    Examples:

    ;;; The following example is from a common-lisp mailing list discussion
    ;;;  (from code suggested by Pavel Curtis).   The question is whether or
    ;;;  not the special declaration in FOO applies to the  (1+ x) form.

    (setf (symbol-value 'x) 6)
    (defun foo (x)				;a lexical binding of X
      (print x)
      (let ((x (1+ x)))			;is the second X special or not?
        (declare (special x))		;`normal' declaration
        (bar))
      (1+ x))
    (defun bar () (print (locally (declare (special x)) x)))

    (foo 10)  will  printout of 10 and 11 by either proposal herein
    (foo 10)  will  printout of 10 and 7 by CLtL style "hoisting"

    ;;; The following example is due to Jim Boyce, of Lucid Inc.  It shows how
    ;;;  the "hoisting" of the declaration inadvertently causes it to act more
    ;;;  like a proclamation than a declaration; namely, the declaration is
    ;;;  applied to two different variables (which happen to have the same
    ;;;  name) -- the first variable is the lexical one bound on line [1] and
    ;;;  the second variables is bound on line [3].  Whereas lexical scoping
    ;;;  rules would say that the reference in line [2] is to the variable
    ;;;  bound on line [1], the effect of the "hoisted" declaration is to
    ;;;  make the line [1]'s variable inaccessible in the initial value forms.

    (setf (symbol-value 'x) 6)

    (defun bar (x y)	    ;[1] 1st instance of x
      (let ((old-x x)	    ;[2] 2nd instance of x -- same as 1st instance?
            (x y))		    ;[3] 3rd instance
        (declare (special x))
        (list old-x x)))

    (bar 'first 'second)	==>  (first second)    ;by either proposal herein
    (bar 'first 'second)	==>  (6 second)        ;by "hoisting", a la CLtL.

    Rationale:

    These semantics are simpler to understand.  Almost everyone feels that
    the example of CLtL p.155 is very unnatural.  LIMITED-HOISTING is less
    of a change to CLtL semantics; but NO-HOISTING seems more natural to
    most people since it restores a closer equivalence between LET forms
    and LAMBDA-expressions.  Also, several vendors report that customers
    frequently seem to assume the semantics of NO-HOISTING.

    Current practice:

    Most implementations implement the rules in CLtL, as exemplified by
    the example on p.155.  Symbolics currently implements rules based on
    Zetalisp which are different from both this proposal and Common Lisp.
    Symbolics plans to change to Common Lisp rules in the future.

    Cost to Implementors:

    Modest; some minor fixes will be necessary to to compilers, interpreters
    and "code walkers" that parse declarations.

    Cost to Users:

    Modest.  It is mostly moot since users tend to avoid the "hoisting"
    situations on special declarations.

    It is possible to mechanically examine a program to determine whether
    its behavior would change under the new rules.  This permits an
    implementation to provide a transition tool to ease conversion to the
    new definition.

    Cost of non-adoption:

    Serious non-portability of code, since not every implementor seems to
    agree on how to read the disputed rules of CLtL pp. 153-157; continuing
    confusion in the user community.

    Performance impact:

    None.

    Benefits:

    Elimination of confusion; increase of portability between implementations.

    Esthetics:

    Simplifies the scoping issue; eliminates special-case scoping rules for
    SPECIAL declarations.

    Discussion:

    Only the SPECIAL declaration has semantic import for CL; both
    proposals specify an incompatible change for this case, to "retract"
    the expansive scope stated or implied in CLtL.  All other declarations
    are considered "advice" to an optimizing compiler, and should have
    no semantic effect on correct programs.  However, programmers making
    use of such declarations may notice a larger difference in the
    NO-HOISTING proposal, since some of their INLINE, OPTIMIZE, TYPE,
    etc. declarations will no longer apply to the initial-value forms.

    One idiom which will be adversely affected by both of these proposals is:
       (let ((*a* *a*))
         ;; rebind *a* to it's "old" value
         (declare (special *a*))
         ...)
    where *a* has not been proclaimed special.  This idiom would likely
    have to be written as:
       (let ((*a* (locally (declare (special *a*)) *a*)))
         ;; rebind *a* to it's "old" value
         (declare (special *a*))
         ...)
    or [preferably!] *a* should be proclaimed special.  Similar idiots
    like this may be in use for LAMBDA-Expressions, or DEFUNs etc.

    On the other hand, the inadvertent "shadowing" which prevents the
    following LET's initial value forms from referencing the input argument
    is handily solved by either proposal herein.  If neither of these
    proposals is not adopted, then the intent of the code for BAR:
      (defun bar (x y)	      ;[1] 1st instance of x
        (let ((old-x x)	      ;[2] 2nd instance of x -- same as 1st instance?
    	  (x y))	      ;[3] 3rd instance
          (declare (special x))
          (list old-x x)))
    would likely have to be expressed by introducing new LET contours:
      (defun bar (x y)	      ;[1] 1st instance of x
        (let ((old-x x))	      ;[2] 2nd instance of x -- same as 1st instance?
          (let ((x y))	      ;[3] 3rd instance
            (declare (special x))
            (list old-x x))))

    The source of additional confusion  has long been that TYPE declarations
    had to be treated differently from all other declarations; this was because
    of the prohibition found on p158 of CLtL.  Given the acceptance of the
    DECLARE-TYPE-FREE proposal, it no longer is necessary to make an exception
    for it, nor to categorize declarations into "pervasive" and "non-pervasive",
    or  "free" and "bound".

    It is not the purpose of this proposal to alter, clarify or in any
    other way bear upon the scoping rules of variables in Common Lisp.
    However, a few reminders about these rules will help one understand
    the above prescription.  Except LET*, PROG*, DO*, LABELS, and MACROLET,
    all the other special forms of CLtL p154 which admit declarations have
    the property that the scope of the name binding does not include any
    initial value form.  As a review of these scopes, note:
      -- for LET, FLET, MULTIPLE-VALUE-BIND, none of the initial value
         forms are included in the variables' (or functions') scope;
      -- for DO-<mumble>-SYMBOLS, the initial value forms are not included,
         but the optional result forms are included;
      -- for DO, DOLIST, and DOTIMES, the initial value forms are not
         included, but the stepper forms and the optional result forms
         are included;
      -- for LET*, PROG*, and DO*, a variable's scope also includes the
         remaining initial value forms, for subsequent variable bindings;
      -- for LABELS and MACROLET, a function name's scope includes all the
         code forms for the functions being defined by the special form
         [a compiler writer must know how not to get into an infinite loop
         of substitutions when there are 'in-line' declarations on these
         mutually recursive names];
      -- for a LAMBDA application, none of the explicit value forms are
         included in the bound variable scoping;  however, the 'initform'
         code (if any) for &optional, &key, and &aux bindings are included
         in the same way as LET* does;
      -- for DEFUN, DEFMACRO, DEFTYPE and DEFSETF follow the rules for
         LAMBDA-Expressions (CLtL, Section 5.2.2, pp. 59-66).

    Remember also that new name bindings "shadow" (after a fashion) any
    higher level binding or declarations.  E.g., presuming that no
    proclamations are in effect, consider the inner let bindings of:
       (locally (declare (special x) (float y))
         (let ((x 5) (y 10))
           (print (+ x y))))
    then x is bound as local (not special); and y is bound with no particular
    type information [because the 'y' being bound is a different variable
    than the 'y' declared float in the outer scope].

    It has been suggested that compilers could be a bit more helpful in
    detecting anomalous bindings, such as in the LET* following:
      (defun bar (x y)
        (let* ((old-x x)
               (x y)
               (new-x x))
          (declare (special x))
          (list old-x x new-x)))
    The collection of variables named X in the LET* binding and initial
    forms includes both local (lexical) and special ones.

Copyright 1996-2005, LispWorks Ltd. All rights reserved.