CLHS: Issue LOOP-INITFORM-ENVIRONMENT Writeup

#

## Issue LOOP-INITFORM-ENVIRONMENT Writeup

    Issue:        LOOP-INITFORM-ENVIRONMENT
    Forum:	      Editorial
    References:   LOOP (pages 8-85,
    	      Draft 8.81
    Category:     CLARIFICATION
    Edit history: 05-Mar-91, Version 1 by Pitman
    	      15-Mar-91, Version 2 by Pitman
    Status:	      For X3J13 consideration

    Problem Description:

      The Symbolics implementation of X3J13's LOOP spec surprised some users
      by being incompatible with the old Symbolics Genera LOOP on the
      following test case:

        (let ((list '(1 2 3)))
          (loop for list = list then (cdr list)
    	 until (null list)
    	 do (princ (car list))))

      The Symbolics Genera implementation prints nothing, but old Symbolics
      Genera LOOP prints 123. In double-checking the implementation against the
      working draft specification to determine if the implementation was in
      error, more than one Symbolics implementors did not feel this behavior
      was clearly enough spelled out and worried that implementations might not
      agree.

      Further investigation showed that Lucid Common Lisp had the same behavior.

      The following are the only references which appeared relevant:

       From page 8-85:

        The LOOP macro translates the given <form> into code and returns the
        expanded <form>.   The expanded <form> is one or more <lambda
        expressions> for the local <binding> of loop variables and a block
        and a tagbody that express a looping control structure. The variables
        established in LOOP are bound as if by LET or LAMBDA.
        Implementations can interleave the setting of initial values with
        the bindings.  However, the assignment of the initial values is
        always calculated in othe order specified by the user.  A variable is
        thus sometimes bound to a meaningless value of the correct type
        and then later in the prologue is set to the true initial value by
        using SETQ.

       Later, on page 8-86:

        The FOR and AS constructs provide iteration control clauses that
        establish a variable to be initialized.  FOR and AS clauses can be
        combined with the LOOP keyword AND to get parallel initialization and
        stepping.  Otherwise, the initialization and stepping are sequential.

      The need for this information is so fundamental that it should be very
      plainly stated.  These extremely vague phrases don't really say much
      about the environment, and since they don't say what goes into the LET
      or the LAMBDA, or even how many LET or LAMBDA forms are involved, they
      don't really say much at all.

      Further, the vague statement on p8-85 about how LET+SETQ might be used to
      implement binding leaves an unusually large amount of latitude to
      implementations.

    Proposal (LOOP-INITFORM-ENVIRONMENT:INITFORM-BEFORE-BINDING)

      Clarify that the initforms in a FOR-AS clause
      (each being variously called the <form1>, the <hash-table>, or the
      <package> in the `bnf' on pages 8-82..8-83)
      are all evaluated (in left to right order) prior to establishing
      the binding for any of the the <vars> in the same clause.

      Clarify that the initforms in a WITH clause (each being
      called a <form1> in the `bnf' on pages 8-82..8-83)
      are all evaluated (in left to right order) prior to establishing
      the binding for any <varN> in the same clause.

    Test Case:

        (let ((list '(1 2 3)))
          (loop for list = list then (cdr list)
    	 until (null list)
    	 collect (car list)))
        => (1 2 3)

    Rationale:

      This is what most users who make an analogy to LET or DO will expect.

    Current Practice:

      Symbolics Genera's FUTURE-COMMON-LISP:LOOP (which purports to conform
      to the draft specification) returns NIL.

      Symbolics Genera's LISP:LOOP (which purports to conform to CLtL)
      returns (1 2 3) for the test case.

    Cost to Implementors:

      Hopefully small.

    Cost to Users:

      Hopefully most users will consider this the status quo.

    Cost of Non-Adoption:

      The specification might be ambiguous on an important point.

    Benefits:

      Cost of non-adoption is avoided.

    Aesthetics:

      Clarity improves aesthetics.

    Discussion:

      Pitman and Moon support this proposal.
      JonL opposes this proposal.

      A long-winded discussion ensued, excerpts of which follow.

      Moon writes:
       ``It's intentional that implementations should have latitude in
         the expansion of LOOP, but not to the extent that the meaning of
         the program changes!  The problem is that the only thing this
         version of the LOOP specification says about the scope of LOOP
         variables is that they are lexical unless proclaimed special and
         their scope is the loop.  This isn't specific enough.

       ``Also in CLtL2 p.722 there is an example expansion which could be
         taken as an argument against the INITFORM-BEFORE-BINDING proposal.
         However I don't think that example was intended to talk about
         variable scoping.  This example does not appear in the draft ANSI
         CL spec, which is good in my opinion.''

      JonL writes:
       ``I don't see this as a clarification but as a change;  in particular,
         it isn't consistent with the section of CLtL/II that you cite:
         [... page 8-85 ...]
         since it forbids the aforementioned "interleaving".  As Moon said,
         it is "intentional that implementations should have latitude in
         the expansion of LOOP, ...".  So the question is just how
         importantant is the case of loop variables "shadowing" local
         lexical variables?  That is, I agree that the current spec is
         ambiguous on just when the binding of 'list' occurs in
    	  (let ((list '(1 2 3)))
    	    (loop for list = list then (cdr list)
    	    . . . ))
         but I'm more inclined to say SO WHAT?  Is the world going into
         terminal meltdown just because this case isn't completely portable?
         is LOOP completely unusable *** just because of this lexical
         shadowing ambiguity?

       ``The point I'm trying to make is not that this case isn't a problem,
         but that it isn't a *big* problem; and even if it remained
         ambiguous (as was the intention for the draft proposal -- reasons
         cited below), it is nowhere near the magnitude of problem that we
         face in other areas.

       ``... Lucid's LOOP (unchanged for many years now) was modeled after
         GSB's MIT code, and it has the same behaviour as Genera.  Even if
         the "original designers" intended something here, I think Glenn's
         execution of it might have led people to think otherwise. ...''

      Moon writes:

       ``I don't see this particular issue as some technical issue of angels
         dancing on the heads of pins that is only of real interest to
         specialists.  It's not uncommon for user programs to use the same
         variable name more than once.  I hear that users very often find it
         frustrating to try to use LOOP because there are implementations around
         that do unintuitive things and documentation around that they can't
         decode.  Those problems are so unnecessary.

       ``... I just now loaded up the MIT LOOP, unchanged for nearly five
         years (unless someone's been changing it without updating the edit
         history at the front) from REAGAN.AI.MIT.EDU, and tried the test case
         from the cleanup issue.  It prints 123, so it conforms to the cleanup.
         This makes me suspect that the MIT LOOP has always had the intuitive
         behavior, but programmers at Lucid and also at Symbolics broke it in the
         process of improving it.  I don't think it's true that Glenn implemented
         it differently from what the cleanup says.

       ``So then I tried the CMU implementation referenced in Scott Fahlman's
         recent message.  It, too, conforms to the cleanup, although it seems to
         have its share of bugs (several compiler warnings and errors while
         loading that I fixed up and proceeded past).

       ``... We can always specify a language feature as doing something
         unpredictable, but is that really necessary?''

      JonL writes:

       ``I claim this ambiguity is purposeful -- the "interleaving" of binding
         and initialization clauses, which in some implementations has
         performance consequences  -- and that the case that brought this
         ambiguity to public attention is not of great consequence to the
         language.

       ``Of course, I agree that we must specify *exactly* the behaviour
         of LET when local variables are being shadowed, such as in:
           (let ((x (list x))) . . .)
         because renaming of bound variables is an important concept (not just
         "pinheads dancing on angels").  But LOOP bindings are not documented to
         be *exactly* like LET, and they aren't the primariy means to achieve
         lexical "shadows"; thus I strongly disagree that seeking out and "fixing"
         documented ambiguities in the binding-order for LOOP "shadows" is of high
         priority.''

Copyright 1996-2005, LispWorks Ltd. All rights reserved.