\start
Date: Sun, 6 Nov 2011 08:11:50 -0600
From: Tim Daly
To: Waldek Hebisch
Subject: Clarification

It appears that the topic of merging systems has hit the fricas mailing list.

Two comments puzzle me.

  "Concerning merge, IMO merging back with Axiom is out of question -- the
   project go in too different directions (one trivial (but easy to 
   understand) example beeing that I spent several months removing trash
   from FriCAS while Axiom at the same time added more trash)."

What "trash" has Axiom added?

  "In FriCAS I deliberately limited changes to Spad language (to maximize
   compatibility with existing code)..."

Actually there is a subtle issue that make FriCAS Spad code non-portable.
There are new lisp functions quietly introduced that only exist in FriCAS
(e.g. GETREFV32) but are referenced in the Spad code. This code compiles 
but fails at runtime. 

There is no mention of this incompatibility in the pamphlet file
(u32vec.spad.pamphlet). The whole reason to use literate programming
is to communicate such facts to other people.

The use of literate programming is likely to be the main reason why
FriCAS (and OpenAxiom) will never merge back. I would like to keep the
algebra compatible.  I do my best to keep up with the new and changed
algebra but it is a bit of a challenge. The new algebra is packaged in
pamphlet files but there is no actual use of the literate style. New
algebra is not explained, there is no connection shown between theory
and its reduction to practice. There are no examples. There are no notes
to explain why new domains have incompatibilities that deviate from the
Spad language and will not compile in Axiom.

In fact, there is no attempt to communicate with the reader at all. 
The files contain raw Spad code. Why do you keep pamphlet files?

\start
Date: Mon, 07 Nov 2011 20:05:08 -0500
From: Leon Baum
To: list
Subject: Newer GCL needed for gcc-4.6?

I've been using my distro's packaged version of FriCAS, but I wanted to
try AXIOM because I like the idea of literate programming.

I ended up getting a github snapshot compiled with gcc-4.5, but just in
case you aren't already aware, I wanted to let you know that the bundled
version of GCL has problems with gcc-4.6.

According to the GCL mailing list [1] and a Debian bug report [2], it
looks like the problems have been already fixed upstream, so simply
updating GCL should fix things in AXIOM.

Error snippet:
...
Compiling gcl_cmpwt.lsp.
Loading ./gcl_cmpinit.lsp
Loading ../lsp/sys-proclaim.lisp
Finished loading ../lsp/sys-proclaim.lisp
Loading sys-proclaim.lisp
Finished loading sys-proclaim.lisp
Finished loading ./gcl_cmpinit.lsp
End of Pass 1.

Error: Someone tried to RETURN-FROM a TAGBODY.
Error signalled by LET.
Backtrace: system::gcl-top-level > if > let > setq > system:error-set > sys=
tem:error-set > progn > compile-file > loop > setq > apply > apply > compil=
e-file1 > let > unwind-protect > progn > let* > let > unwind-protect > prog=
n > when > compiler-pass2 > let > unwind-protect > progn > let > let > unwi=
nd-protect > progn > catch > ctop-write > do* > when > apply > apply > t3de=
fun > cond > analyze-regs > let > cond > let > analyze-regs1 > let > tagbod=
y > system:universal-error-handler > let > let > cond > apply > apply > sys=
tem:universal-error-handler > system::break-level > let* > UNLESS
gcl_cmpwt.c: In function =E2=80=98init_gcl_cmpwt=E2=80=99:
gcl_cmpwt.c:4:39: error: =E2=80=98VV=E2=80=99 undeclared (first use in this=
 function)
...


Leon

[1] http://lists.gnu.org/archive/html/gcl-devel/2011-04/msg00005.html
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D625032

\start
Date: Mon, 07 Nov 2011 20:11:08 -0500
From: Tim Daly
To: Leon Baum
Subject: Re: Newer GCL needed for gcc-4.6?

Sigh. Thanks for the message.
I'll update the GCL version as soon as I can.
What linux distro are you running?

Tim Daly


\start
Date: Mon, 7 Nov 2011 21:51:04 -0600
From: Tim Daly
To: Camm Maguire
Subject: [GCL] EXTRAS are not included

I checked out the latest gcl-2.6.8pre.

I have a patch that gets applied to h/linux.defs which adds the line:

EXTRAS = ${OBJ}/${SYS}/lib/cfuns-c.o ${OBJ}/${SYS}/lib/sockio-c.o

This had the effect of linking these two files into the lisp image.
Unfortunately the files don't appear to be linked. The OBJ and SYS
variables exist and have always worked in the past. The cfun-c.o
and sockio-c.o files exist as well as cfuns-c.ini and sockio-c.ini

cfuns-c defines a function called writeablep but this is not linked
into the image.

I don't see any changes related to EXTRAS.

What am I missing?

\start
Date: Tue, 8 Nov 2011 08:25:24 -0600
From: Tim Daly
To: list
Subject: Literate Programming -- Knuth interview

Knuth said:

Literate programming is a very personal thing. I think it's terrific,
but that might well be because I'm a very strange person. It has tens
of thousands of fans, but not millions.
 
In my experience, software created with literate programming has
turned out to be significantly better than software developed in more
traditional ways. Yet ordinary software is usually okay -- I'd give it
a grade of C (or maybe C++), but not F; hence, the traditinal methods
stay with us. Since they're understood by a vast community of
programmers, most people have no big incentive to change, just as I'm
not motivated to learn Esperanto even though it might be preferable to
English and German and French and Russian (if everybody switched).

Jon Bentley probably hit the nail on the head when he once was asked
why literate programming hasn't taken the whole world by storm. He
observed that a small percentage of the world's population is good at
programming, and a small percentage is good at writing; apparently I
am asking everybody to be in both subsets.

Yet to me, literate programming is certainly the most important thing
that came out of the TeX project. Not only has it enabled me to write
and maintain programs faster and more reliably than ever before, and
been one of my greatest sources of joy since the 1980s -- it has
actually been indispensable at times. Some of my major programs, such
as the MMIX meta-simulator, could not have been written with any other
methodology that I've ever heard of. The complexity was simply too
daunting for my limited brain to handle; without literate programming,
the whole enterprise would have flopped miserably.

If people discover nice ways to use the newfangled multithreaded
machines, I would expect the discovery to come from people who
routinely use literate programming. Literate programming is what you
need to rise above the ordinary level of achievement. But I don't
believe in forcing ideas on anybody. If literate programming isn't
your style, please forget it and do what you like. If nobody likes it
but me, let it die.

Knuth, Donald ``Interview with Donald Knuth''
www.informit.com/articles/article.aspx?p=1193856&_

\start
Date: Tue, 8 Nov 2011 08:46:19 -0600
From: Tim Daly
To: list
Subject: Literate Programming -- Knuth interview

> Yet to me, literate programming is certainly the most important thing
> that came out of the TeX project. Not only has it enabled me to write
> and maintain programs faster and more reliably than ever before, and
> been one of my greatest sources of joy since the 1980s -- it has
> actually been indispensable at times. Some of my major programs, such
> as the MMIX meta-simulator, could not have been written with any other
> methodology that I've ever heard of. The complexity was simply too
> daunting for my limited brain to handle; without literate programming,
> the whole enterprise would have flopped miserably.
> 
> If people discover nice ways to use the newfangled multithreaded
> machines, I would expect the discovery to come from people who
> routinely use literate programming. Literate programming is what you
> need to rise above the ordinary level of achievement. 

I believe that Axiom's complexity is large enough to demand literate
programming. There are several reasons.

First, computational mathematics requires people to be exceptional at
mathematics and programming. This is a small subset of already small
sets. We might as well add another subset of those who can communicate
their ideas in writing.

Second, there are many design decisions that are necessary to reduce
a mathematical idea to implementation. Some of these design decisions
are mathematically arbitrary (e.g. branch cuts) or computationally
arbitrary (e.g. sparse versus dense) or programatically arbitrary
(e.g. all Spad versus Spad-and-lisp). These design decisions need to
be documented so people who maintain and modify the program know why
the decisions were made. Without this knowledge it would be trivial
to accidently destroy important optimizations.

Third, nobody is an expert in the range of mathematics that Axiom can
and will implement. It is important to present some portions of the
theory associated with domains so people have a clue about the ideas
being encoded. Imagine what would happen if all of the math textbooks
only contained equations but no human-readable text. You might be able
to "read" a calculus textbook but not an infinite group theory textbook.

Fourth, a million line program is too large to put into your head. You
need to have some background on the data structures, control flow, any
special tricks (e.g. funcalls through the symbol-plist), database
design, communication protocols (e.g. between Axiom and Hyperdoc and
Graphics), parsing structures, and a million other details. At best,
the code tells you HOW it does something but not WHY, not what it
depends on, not what depends on it, etc.

Fifth, a program this large and this long-lived will eventually no
longer have the authors around to ask questions. Several of the Axiom
authors are already dead and most are not associated with it anymore.
Some of those authors are the world's expert in their subject matter.
That is a steep hill to climb if you want to understand the code,
especially if you have to debug or modify it.

Sixth, in the 30 year horizon view, we need to pursue a new level of
excellence. As Knuth said:

   "Literate programming is what you need to rise above 
    the ordinary level of achievement"

For all these reasons, and more, Axiom needs to be literate.

\start
Date: Tue, 08 Nov 2011 15:03:36 -0500
From: Eugene Surowitz
To: list
Subject: Re: Literate Programming -- Knuth interview

That literate programming is fully justified for Axiom is, well,
almost axiomatic.  But the issue is more how to boost the ability to
invert the process and reverse engineer non-literate code piles into
literate documents.

What, in your opinion, would be the most effective type of tool that could
be developed to render the literacy project more tractable?

Eugene J. Surowitz

On 11/8/2011 9:46 AM, Tim Daly wrote:
>> Yet to me, literate programming is certainly the most important thing
>> that came out of the TeX project. Not only has it enabled me to write
>> and maintain programs faster and more reliably than ever before, and
>> been one of my greatest sources of joy since the 1980s -- it has
>> actually been indispensable at times. Some of my major programs, such
>> as the MMIX meta-simulator, could not have been written with any other
>> methodology that I've ever heard of. The complexity was simply too
>> daunting for my limited brain to handle; without literate programming,
>> the whole enterprise would have flopped miserably.
>>
>> If people discover nice ways to use the newfangled multithreaded
>> machines, I would expect the discovery to come from people who
>> routinely use literate programming. Literate programming is what you
>> need to rise above the ordinary level of achievement.
>
> I believe that Axiom's complexity is large enough to demand literate
> programming. There are several reasons.
>
> First, computational mathematics requires people to be exceptional at
> mathematics and programming. This is a small subset of already small
> sets. We might as well add another subset of those who can communicate
> their ideas in writing.
>
> Second, there are many design decisions that are necessary to reduce
> a mathematical idea to implementation. Some of these design decisions
> are mathematically arbitrary (e.g. branch cuts) or computationally
> arbitrary (e.g. sparse versus dense) or programatically arbitrary
> (e.g. all Spad versus Spad-and-lisp). These design decisions need to
> be documented so people who maintain and modify the program know why
> the decisions were made. Without this knowledge it would be trivial
> to accidently destroy important optimizations.
>
> Third, nobody is an expert in the range of mathematics that Axiom can
> and will implement. It is important to present some portions of the
> theory associated with domains so people have a clue about the ideas
> being encoded. Imagine what would happen if all of the math textbooks
> only contained equations but no human-readable text. You might be able
> to "read" a calculus textbook but not an infinite group theory textbook.
>
> Fourth, a million line program is too large to put into your head. You
> need to have some background on the data structures, control flow, any
> special tricks (e.g. funcalls through the symbol-plist), database
> design, communication protocols (e.g. between Axiom and Hyperdoc and
> Graphics), parsing structures, and a million other details. At best,
> the code tells you HOW it does something but not WHY, not what it
> depends on, not what depends on it, etc.
>
> Fifth, a program this large and this long-lived will eventually no
> longer have the authors around to ask questions. Several of the Axiom
> authors are already dead and most are not associated with it anymore.
> Some of those authors are the world's expert in their subject matter.
> That is a steep hill to climb if you want to understand the code,
> especially if you have to debug or modify it.
>
> Sixth, in the 30 year horizon view, we need to pursue a new level of
> excellence. As Knuth said:
>
>     "Literate programming is what you need to rise above
>      the ordinary level of achievement"
>
> For all these reasons, and more, Axiom needs to be literate.

\start
Date: Tue, 08 Nov 2011 16:42:05 -0500
From: Tim Daly
To: Eugene Surowitz
Subject: Re: Literate Programming -- Knuth interview

Tools? The goal of literate programming is communicating from
human to human. It is like writing a novel. All you need is a
working Underwood typewriter and time.

I tend to favor Latex because a lot of math is involved and
Latex prints math well. But I'm presenting a talk about
Literate Programming in North Carolina this week and I plan
to show examples that use XML, html, and video.

Tim

On Tue, 2011-11-08 at 15:03 -0500, Eugene Surowitz wrote:
> That literate programming is fully justified for Axiom is, well, almost axiomatic.
> But the issue is more how to boost the ability to invert the process
> and reverse engineer non-literate code piles into literate documents.
> 
> What, in your opinion, would be the most effective type of tool that could
> be developed to render the literacy project more tractable?
> 
> Eugene J. Surowitz
> 
> On 11/8/2011 9:46 AM, Tim Daly wrote:
> >> Yet to me, literate programming is certainly the most important thing
> >> that came out of the TeX project. Not only has it enabled me to write
> >> and maintain programs faster and more reliably than ever before, and
> >> been one of my greatest sources of joy since the 1980s -- it has
> >> actually been indispensable at times. Some of my major programs, such
> >> as the MMIX meta-simulator, could not have been written with any other
> >> methodology that I've ever heard of. The complexity was simply too
> >> daunting for my limited brain to handle; without literate programming,
> >> the whole enterprise would have flopped miserably.
> >>
> >> If people discover nice ways to use the newfangled multithreaded
> >> machines, I would expect the discovery to come from people who
> >> routinely use literate programming. Literate programming is what you
> >> need to rise above the ordinary level of achievement.
> >
> > I believe that Axiom's complexity is large enough to demand literate
> > programming. There are several reasons.
> >
> > First, computational mathematics requires people to be exceptional at
> > mathematics and programming. This is a small subset of already small
> > sets. We might as well add another subset of those who can communicate
> > their ideas in writing.
> >
> > Second, there are many design decisions that are necessary to reduce
> > a mathematical idea to implementation. Some of these design decisions
> > are mathematically arbitrary (e.g. branch cuts) or computationally
> > arbitrary (e.g. sparse versus dense) or programatically arbitrary
> > (e.g. all Spad versus Spad-and-lisp). These design decisions need to
> > be documented so people who maintain and modify the program know why
> > the decisions were made. Without this knowledge it would be trivial
> > to accidently destroy important optimizations.
> >
> > Third, nobody is an expert in the range of mathematics that Axiom can
> > and will implement. It is important to present some portions of the
> > theory associated with domains so people have a clue about the ideas
> > being encoded. Imagine what would happen if all of the math textbooks
> > only contained equations but no human-readable text. You might be able
> > to "read" a calculus textbook but not an infinite group theory textbook.
> >
> > Fourth, a million line program is too large to put into your head. You
> > need to have some background on the data structures, control flow, any
> > special tricks (e.g. funcalls through the symbol-plist), database
> > design, communication protocols (e.g. between Axiom and Hyperdoc and
> > Graphics), parsing structures, and a million other details. At best,
> > the code tells you HOW it does something but not WHY, not what it
> > depends on, not what depends on it, etc.
> >
> > Fifth, a program this large and this long-lived will eventually no
> > longer have the authors around to ask questions. Several of the Axiom
> > authors are already dead and most are not associated with it anymore.
> > Some of those authors are the world's expert in their subject matter.
> > That is a steep hill to climb if you want to understand the code,
> > especially if you have to debug or modify it.
> >
> > Sixth, in the 30 year horizon view, we need to pursue a new level of
> > excellence. As Knuth said:
> >
> >     "Literate programming is what you need to rise above
> >      the ordinary level of achievement"
> >
> > For all these reasons, and more, Axiom needs to be literate.

\start
Date: Fri, 11 Nov 2011 11:03:26 -0500
From: Eugene Surowitz
To: Tim Daly
Subject: Re: Literate Programming -- Knuth interview

I prefer the fountain pen for novelistic efforts.

The issue that aroused my question has to do with
comments that appear now and then in the list
about knowing what is where in the source code.

The reference to tools really was intended to address
the idea of the utility of a mechanism(s) that could
globally treat the entire source code as text to
answer that question.

On 11/8/2011 4:42 PM, daly wrote:
> Tools? The goal of literate programming is communicating from
> human to human. It is like writing a novel. All you need is a
> working Underwood typewriter and time.
>
> I tend to favor Latex because a lot of math is involved and
> Latex prints math well. But I'm presenting a talk about
> Literate Programming in North Carolina this week and I plan
> to show examples that use XML, html, and video.
>
> Tim
>
> On Tue, 2011-11-08 at 15:03 -0500, Eugene Surowitz wrote:
>> That literate programming is fully justified for Axiom is, well, almost axiomatic.
>> But the issue is more how to boost the ability to invert the process
>> and reverse engineer non-literate code piles into literate documents.
>>
>> What, in your opinion, would be the most effective type of tool that could
>> be developed to render the literacy project more tractable?
>>
>> Eugene J. Surowitz
>>
>> On 11/8/2011 9:46 AM, Tim Daly wrote:
>>>> Yet to me, literate programming is certainly the most important thing
>>>> that came out of the TeX project. Not only has it enabled me to write
>>>> and maintain programs faster and more reliably than ever before, and
>>>> been one of my greatest sources of joy since the 1980s -- it has
>>>> actually been indispensable at times. Some of my major programs, such
>>>> as the MMIX meta-simulator, could not have been written with any other
>>>> methodology that I've ever heard of. The complexity was simply too
>>>> daunting for my limited brain to handle; without literate programming,
>>>> the whole enterprise would have flopped miserably.
>>>>
>>>> If people discover nice ways to use the newfangled multithreaded
>>>> machines, I would expect the discovery to come from people who
>>>> routinely use literate programming. Literate programming is what you
>>>> need to rise above the ordinary level of achievement.
>>>
>>> I believe that Axiom's complexity is large enough to demand literate
>>> programming. There are several reasons.
>>>
>>> First, computational mathematics requires people to be exceptional at
>>> mathematics and programming. This is a small subset of already small
>>> sets. We might as well add another subset of those who can communicate
>>> their ideas in writing.
>>>
>>> Second, there are many design decisions that are necessary to reduce
>>> a mathematical idea to implementation. Some of these design decisions
>>> are mathematically arbitrary (e.g. branch cuts) or computationally
>>> arbitrary (e.g. sparse versus dense) or programatically arbitrary
>>> (e.g. all Spad versus Spad-and-lisp). These design decisions need to
>>> be documented so people who maintain and modify the program know why
>>> the decisions were made. Without this knowledge it would be trivial
>>> to accidently destroy important optimizations.
>>>
>>> Third, nobody is an expert in the range of mathematics that Axiom can
>>> and will implement. It is important to present some portions of the
>>> theory associated with domains so people have a clue about the ideas
>>> being encoded. Imagine what would happen if all of the math textbooks
>>> only contained equations but no human-readable text. You might be able
>>> to "read" a calculus textbook but not an infinite group theory textbook.
>>>
>>> Fourth, a million line program is too large to put into your head. You
>>> need to have some background on the data structures, control flow, any
>>> special tricks (e.g. funcalls through the symbol-plist), database
>>> design, communication protocols (e.g. between Axiom and Hyperdoc and
>>> Graphics), parsing structures, and a million other details. At best,
>>> the code tells you HOW it does something but not WHY, not what it
>>> depends on, not what depends on it, etc.
>>>
>>> Fifth, a program this large and this long-lived will eventually no
>>> longer have the authors around to ask questions. Several of the Axiom
>>> authors are already dead and most are not associated with it anymore.
>>> Some of those authors are the world's expert in their subject matter.
>>> That is a steep hill to climb if you want to understand the code,
>>> especially if you have to debug or modify it.
>>>
>>> Sixth, in the 30 year horizon view, we need to pursue a new level of
>>> excellence. As Knuth said:
>>>
>>>      "Literate programming is what you need to rise above
>>>       the ordinary level of achievement"
>>>
>>> For all these reasons, and more, Axiom needs to be literate.

\start
Date: Mon, 14 Nov 2011 14:08:37 -0500
From: Eugene Surowitz
To: Tim Daly
Subject: Re: Literate Programming -- Knuth interview

Will your talk and slides be available on the web?

Long term objective excellent; getting there is the rub.
It's ok at the developed and distributed level,
but from the developer's ant eye view the toc and index
are only as good as the choice of structure and
the entry selection.

There is also the headache of what the eyeball missed
and what opinion bias does or does not consider important.
Also texts tend to have concepts expressed as multiword
strings, such as "table of contents".
Pseudo-letters, such as "_" or "~", just add to the mess.

Exhaustive listing of everything was quite useful
in another book effort.  For a code the size of Axiom,
I guesstimate that a factor of three for the token file size.

On 11/12/2011 9:16 PM, daly wrote:
> Well, the question is whether you're looking for a concept
> (check table of contents) or a particular function (check
> the index). At least that is the long term intention.
>
> On Fri, 2011-11-11 at 11:03 -0500, Eugene Surowitz wrote:
>> I prefer the fountain pen for novelistic efforts.
>>
>> The issue that aroused my question has to do with
>> comments that appear now and then in the list
>> about knowing what is where in the source code.
>>
>> The reference to tools really was intended to address
>> the idea of the utility of a mechanism(s) that could
>> globally treat the entire source code as text to
>> answer that question.
>>
>> Eugene J. Surowitz
>>
>> On 11/8/2011 4:42 PM, daly wrote:
>>> Tools? The goal of literate programming is communicating from
>>> human to human. It is like writing a novel. All you need is a
>>> working Underwood typewriter and time.
>>>
>>> I tend to favor Latex because a lot of math is involved and
>>> Latex prints math well. But I'm presenting a talk about
>>> Literate Programming in North Carolina this week and I plan
>>> to show examples that use XML, html, and video.
>>>
>>> Tim
>>>
>>> On Tue, 2011-11-08 at 15:03 -0500, Eugene Surowitz wrote:
>>>> That literate programming is fully justified for Axiom is, well, almost axiomatic.
>>>> But the issue is more how to boost the ability to invert the process
>>>> and reverse engineer non-literate code piles into literate documents.
>>>>
>>>> What, in your opinion, would be the most effective type of tool that could
>>>> be developed to render the literacy project more tractable?
>>>>
>>>> Eugene J. Surowitz
>>>>
>>>> On 11/8/2011 9:46 AM, Tim Daly wrote:
>>>>>> Yet to me, literate programming is certainly the most important thing
>>>>>> that came out of the TeX project. Not only has it enabled me to write
>>>>>> and maintain programs faster and more reliably than ever before, and
>>>>>> been one of my greatest sources of joy since the 1980s -- it has
>>>>>> actually been indispensable at times. Some of my major programs, such
>>>>>> as the MMIX meta-simulator, could not have been written with any other
>>>>>> methodology that I've ever heard of. The complexity was simply too
>>>>>> daunting for my limited brain to handle; without literate programming,
>>>>>> the whole enterprise would have flopped miserably.
>>>>>>
>>>>>> If people discover nice ways to use the newfangled multithreaded
>>>>>> machines, I would expect the discovery to come from people who
>>>>>> routinely use literate programming. Literate programming is what you
>>>>>> need to rise above the ordinary level of achievement.
>>>>>
>>>>> I believe that Axiom's complexity is large enough to demand literate
>>>>> programming. There are several reasons.
>>>>>
>>>>> First, computational mathematics requires people to be exceptional at
>>>>> mathematics and programming. This is a small subset of already small
>>>>> sets. We might as well add another subset of those who can communicate
>>>>> their ideas in writing.
>>>>>
>>>>> Second, there are many design decisions that are necessary to reduce
>>>>> a mathematical idea to implementation. Some of these design decisions
>>>>> are mathematically arbitrary (e.g. branch cuts) or computationally
>>>>> arbitrary (e.g. sparse versus dense) or programatically arbitrary
>>>>> (e.g. all Spad versus Spad-and-lisp). These design decisions need to
>>>>> be documented so people who maintain and modify the program know why
>>>>> the decisions were made. Without this knowledge it would be trivial
>>>>> to accidently destroy important optimizations.
>>>>>
>>>>> Third, nobody is an expert in the range of mathematics that Axiom can
>>>>> and will implement. It is important to present some portions of the
>>>>> theory associated with domains so people have a clue about the ideas
>>>>> being encoded. Imagine what would happen if all of the math textbooks
>>>>> only contained equations but no human-readable text. You might be able
>>>>> to "read" a calculus textbook but not an infinite group theory textbook.
>>>>>
>>>>> Fourth, a million line program is too large to put into your head. You
>>>>> need to have some background on the data structures, control flow, any
>>>>> special tricks (e.g. funcalls through the symbol-plist), database
>>>>> design, communication protocols (e.g. between Axiom and Hyperdoc and
>>>>> Graphics), parsing structures, and a million other details. At best,
>>>>> the code tells you HOW it does something but not WHY, not what it
>>>>> depends on, not what depends on it, etc.
>>>>>
>>>>> Fifth, a program this large and this long-lived will eventually no
>>>>> longer have the authors around to ask questions. Several of the Axiom
>>>>> authors are already dead and most are not associated with it anymore.
>>>>> Some of those authors are the world's expert in their subject matter.
>>>>> That is a steep hill to climb if you want to understand the code,
>>>>> especially if you have to debug or modify it.
>>>>>
>>>>> Sixth, in the 30 year horizon view, we need to pursue a new level of
>>>>> excellence. As Knuth said:
>>>>>
>>>>>       "Literate programming is what you need to rise above
>>>>>        the ordinary level of achievement"
>>>>>
>>>>> For all these reasons, and more, Axiom needs to be literate.

\start
Date: Mon, 14 Nov 2011 19:12:12 -0500
From: Tim Daly
To: Eugene Surowitz
Subject: Re: Literate Programming -- Knuth interview

On Mon, 2011-11-14 at 14:08 -0500, Eugene Surowitz wrote:
> Will your talk and slides be available on the web?

I gave a presentation at the Clojure Conj on Literate
Programming which was fairly well received but there were
no slides. I don't believe the talk was recorded.

I have an article in the Notices of the American Mathematical
Society scheduled to appear in February.

> 
> Long term objective excellent; getting there is the rub.
> It's ok at the developed and distributed level,
> but from the developer's ant eye view the toc and index
> are only as good as the choice of structure and
> the entry selection.

Indeed, that's a struggle for all authors. However, I have
made the index and cross references for every function and
variable.

> 
> There is also the headache of what the eyeball missed
> and what opinion bias does or does not consider important.
> Also texts tend to have concepts expressed as multiword
> strings, such as "table of contents".
> Pseudo-letters, such as "_" or "~", just add to the mess.

Every function, macro, and variable is introduced with
     \defun{foo}
     \defmacro{foo}
     \defvar{foo}

All of the index entries are hand generated using latex as in:
     \calls{foo}{bar}
which will put an entry under "foo" for "bar" and
an entry under "bar" for "foo". This means that it is trivial
to find every place that "bar" is called. This reverse-lookup
usually requires a code walker.

For global variables there are:
     \refsdollar{foo}{var}
     \defsdollar{foo}{var}
which will put an entry in the index for $var.


> 
> Exhaustive listing of everything was quite useful
> in another book effort.  For a code the size of Axiom,
> I guesstimate that a factor of three for the token file size.
> 
> Eugene J. Surowitz
> 
> On 11/12/2011 9:16 PM, daly wrote:
> > Well, the question is whether you're looking for a concept
> > (check table of contents) or a particular function (check
> > the index). At least that is the long term intention.
> >
> > On Fri, 2011-11-11 at 11:03 -0500, Eugene Surowitz wrote:
> >> I prefer the fountain pen for novelistic efforts.
> >>
> >> The issue that aroused my question has to do with
> >> comments that appear now and then in the list
> >> about knowing what is where in the source code.
> >>
> >> The reference to tools really was intended to address
> >> the idea of the utility of a mechanism(s) that could
> >> globally treat the entire source code as text to
> >> answer that question.
> >>
> >> Eugene J. Surowitz
> >>
> >> On 11/8/2011 4:42 PM, daly wrote:
> >>> Tools? The goal of literate programming is communicating from
> >>> human to human. It is like writing a novel. All you need is a
> >>> working Underwood typewriter and time.
> >>>
> >>> I tend to favor Latex because a lot of math is involved and
> >>> Latex prints math well. But I'm presenting a talk about
> >>> Literate Programming in North Carolina this week and I plan
> >>> to show examples that use XML, html, and video.
> >>>
> >>> Tim
> >>>
> >>> On Tue, 2011-11-08 at 15:03 -0500, Eugene Surowitz wrote:
> >>>> That literate programming is fully justified for Axiom is, well, almost axiomatic.
> >>>> But the issue is more how to boost the ability to invert the process
> >>>> and reverse engineer non-literate code piles into literate documents.
> >>>>
> >>>> What, in your opinion, would be the most effective type of tool that could
> >>>> be developed to render the literacy project more tractable?
> >>>>
> >>>> Eugene J. Surowitz
> >>>>
> >>>> On 11/8/2011 9:46 AM, Tim Daly wrote:
> >>>>>> Yet to me, literate programming is certainly the most important thing
> >>>>>> that came out of the TeX project. Not only has it enabled me to write
> >>>>>> and maintain programs faster and more reliably than ever before, and
> >>>>>> been one of my greatest sources of joy since the 1980s -- it has
> >>>>>> actually been indispensable at times. Some of my major programs, such
> >>>>>> as the MMIX meta-simulator, could not have been written with any other
> >>>>>> methodology that I've ever heard of. The complexity was simply too
> >>>>>> daunting for my limited brain to handle; without literate programming,
> >>>>>> the whole enterprise would have flopped miserably.
> >>>>>>
> >>>>>> If people discover nice ways to use the newfangled multithreaded
> >>>>>> machines, I would expect the discovery to come from people who
> >>>>>> routinely use literate programming. Literate programming is what you
> >>>>>> need to rise above the ordinary level of achievement.
> >>>>>
> >>>>> I believe that Axiom's complexity is large enough to demand literate
> >>>>> programming. There are several reasons.
> >>>>>
> >>>>> First, computational mathematics requires people to be exceptional at
> >>>>> mathematics and programming. This is a small subset of already small
> >>>>> sets. We might as well add another subset of those who can communicate
> >>>>> their ideas in writing.
> >>>>>
> >>>>> Second, there are many design decisions that are necessary to reduce
> >>>>> a mathematical idea to implementation. Some of these design decisions
> >>>>> are mathematically arbitrary (e.g. branch cuts) or computationally
> >>>>> arbitrary (e.g. sparse versus dense) or programatically arbitrary
> >>>>> (e.g. all Spad versus Spad-and-lisp). These design decisions need to
> >>>>> be documented so people who maintain and modify the program know why
> >>>>> the decisions were made. Without this knowledge it would be trivial
> >>>>> to accidently destroy important optimizations.
> >>>>>
> >>>>> Third, nobody is an expert in the range of mathematics that Axiom can
> >>>>> and will implement. It is important to present some portions of the
> >>>>> theory associated with domains so people have a clue about the ideas
> >>>>> being encoded. Imagine what would happen if all of the math textbooks
> >>>>> only contained equations but no human-readable text. You might be able
> >>>>> to "read" a calculus textbook but not an infinite group theory textbook.
> >>>>>
> >>>>> Fourth, a million line program is too large to put into your head. You
> >>>>> need to have some background on the data structures, control flow, any
> >>>>> special tricks (e.g. funcalls through the symbol-plist), database
> >>>>> design, communication protocols (e.g. between Axiom and Hyperdoc and
> >>>>> Graphics), parsing structures, and a million other details. At best,
> >>>>> the code tells you HOW it does something but not WHY, not what it
> >>>>> depends on, not what depends on it, etc.
> >>>>>
> >>>>> Fifth, a program this large and this long-lived will eventually no
> >>>>> longer have the authors around to ask questions. Several of the Axiom
> >>>>> authors are already dead and most are not associated with it anymore.
> >>>>> Some of those authors are the world's expert in their subject matter.
> >>>>> That is a steep hill to climb if you want to understand the code,
> >>>>> especially if you have to debug or modify it.
> >>>>>
> >>>>> Sixth, in the 30 year horizon view, we need to pursue a new level of
> >>>>> excellence. As Knuth said:
> >>>>>
> >>>>>       "Literate programming is what you need to rise above
> >>>>>        the ordinary level of achievement"
> >>>>>
> >>>>> For all these reasons, and more, Axiom needs to be literate.

\start
Date: Fri, 18 Nov 2011 12:53:39 -0500
From: Eugene Surowitz
To: Tim Daly
Subject: Re: Literate Programming -- Knuth interview

I was just at BICA2011 and they both record the talks and
are actually creating DVDs along with the collected slides.
(They sold last year's DVDs at this years conference.)
Some are talks are on the web site.

I'm thinking along the line of a "text crawler" complemented
by a relational data base with the capability to call up
the literate pamphlets at a click.  Thus reverse walking
gives a product support and development mechanism.
The net result would be the ability to locate "foobar"
whatever context it is in - code or description.
I am assuming the however well Axiom gets pamphleteer-ed (ouch)
the perversity of human nature will result in individuals
shoehorning and shotgunning in code and comments in when
and where it occurs to them.
I would hope that the easier it is to do literate code cleanly
would discourage that phenomena.

On 11/14/2011 7:12 PM, daly wrote:
> On Mon, 2011-11-14 at 14:08 -0500, Eugene Surowitz wrote:
>> Will your talk and slides be available on the web?
>
> I gave a presentation at the Clojure Conj on Literate
> Programming which was fairly well received but there were
> no slides. I don't believe the talk was recorded.
>
> I have an article in the Notices of the American Mathematical
> Society scheduled to appear in February.
>
>>
>> Long term objective excellent; getting there is the rub.
>> It's ok at the developed and distributed level,
>> but from the developer's ant eye view the toc and index
>> are only as good as the choice of structure and
>> the entry selection.
>
> Indeed, that's a struggle for all authors. However, I have
> made the index and cross references for every function and
> variable.
>
>>
>> There is also the headache of what the eyeball missed
>> and what opinion bias does or does not consider important.
>> Also texts tend to have concepts expressed as multiword
>> strings, such as "table of contents".
>> Pseudo-letters, such as "_" or "~", just add to the mess.
>
> Every function, macro, and variable is introduced with
>       \defun{foo}
>       \defmacro{foo}
>       \defvar{foo}
>
> All of the index entries are hand generated using latex as in:
>       \calls{foo}{bar}
> which will put an entry under "foo" for "bar" and
> an entry under "bar" for "foo". This means that it is trivial
> to find every place that "bar" is called. This reverse-lookup
> usually requires a code walker.
>
> For global variables there are:
>       \refsdollar{foo}{var}
>       \defsdollar{foo}{var}
> which will put an entry in the index for $var.
>
>
>>
>> Exhaustive listing of everything was quite useful
>> in another book effort.  For a code the size of Axiom,
>> I guesstimate that a factor of three for the token file size.
>>
>> Eugene J. Surowitz
>>
>> On 11/12/2011 9:16 PM, daly wrote:
>>> Well, the question is whether you're looking for a concept
>>> (check table of contents) or a particular function (check
>>> the index). At least that is the long term intention.
>>>
>>> On Fri, 2011-11-11 at 11:03 -0500, Eugene Surowitz wrote:
>>>> I prefer the fountain pen for novelistic efforts.
>>>>
>>>> The issue that aroused my question has to do with
>>>> comments that appear now and then in the list
>>>> about knowing what is where in the source code.
>>>>
>>>> The reference to tools really was intended to address
>>>> the idea of the utility of a mechanism(s) that could
>>>> globally treat the entire source code as text to
>>>> answer that question.
>>>>
>>>> Eugene J. Surowitz
>>>>
>>>> On 11/8/2011 4:42 PM, daly wrote:
>>>>> Tools? The goal of literate programming is communicating from
>>>>> human to human. It is like writing a novel. All you need is a
>>>>> working Underwood typewriter and time.
>>>>>
>>>>> I tend to favor Latex because a lot of math is involved and
>>>>> Latex prints math well. But I'm presenting a talk about
>>>>> Literate Programming in North Carolina this week and I plan
>>>>> to show examples that use XML, html, and video.
>>>>>
>>>>> Tim
>>>>>
>>>>> On Tue, 2011-11-08 at 15:03 -0500, Eugene Surowitz wrote:
>>>>>> That literate programming is fully justified for Axiom is, well, almost axiomatic.
>>>>>> But the issue is more how to boost the ability to invert the process
>>>>>> and reverse engineer non-literate code piles into literate documents.
>>>>>>
>>>>>> What, in your opinion, would be the most effective type of tool that could
>>>>>> be developed to render the literacy project more tractable?
>>>>>>
>>>>>> Eugene J. Surowitz
>>>>>>
>>>>>> On 11/8/2011 9:46 AM, Tim Daly wrote:
>>>>>>>> Yet to me, literate programming is certainly the most important thing
>>>>>>>> that came out of the TeX project. Not only has it enabled me to write
>>>>>>>> and maintain programs faster and more reliably than ever before, and
>>>>>>>> been one of my greatest sources of joy since the 1980s -- it has
>>>>>>>> actually been indispensable at times. Some of my major programs, such
>>>>>>>> as the MMIX meta-simulator, could not have been written with any other
>>>>>>>> methodology that I've ever heard of. The complexity was simply too
>>>>>>>> daunting for my limited brain to handle; without literate programming,
>>>>>>>> the whole enterprise would have flopped miserably.
>>>>>>>>
>>>>>>>> If people discover nice ways to use the newfangled multithreaded
>>>>>>>> machines, I would expect the discovery to come from people who
>>>>>>>> routinely use literate programming. Literate programming is what you
>>>>>>>> need to rise above the ordinary level of achievement.
>>>>>>>
>>>>>>> I believe that Axiom's complexity is large enough to demand literate
>>>>>>> programming. There are several reasons.
>>>>>>>
>>>>>>> First, computational mathematics requires people to be exceptional at
>>>>>>> mathematics and programming. This is a small subset of already small
>>>>>>> sets. We might as well add another subset of those who can communicate
>>>>>>> their ideas in writing.
>>>>>>>
>>>>>>> Second, there are many design decisions that are necessary to reduce
>>>>>>> a mathematical idea to implementation. Some of these design decisions
>>>>>>> are mathematically arbitrary (e.g. branch cuts) or computationally
>>>>>>> arbitrary (e.g. sparse versus dense) or programatically arbitrary
>>>>>>> (e.g. all Spad versus Spad-and-lisp). These design decisions need to
>>>>>>> be documented so people who maintain and modify the program know why
>>>>>>> the decisions were made. Without this knowledge it would be trivial
>>>>>>> to accidently destroy important optimizations.
>>>>>>>
>>>>>>> Third, nobody is an expert in the range of mathematics that Axiom can
>>>>>>> and will implement. It is important to present some portions of the
>>>>>>> theory associated with domains so people have a clue about the ideas
>>>>>>> being encoded. Imagine what would happen if all of the math textbooks
>>>>>>> only contained equations but no human-readable text. You might be able
>>>>>>> to "read" a calculus textbook but not an infinite group theory textbook.
>>>>>>>
>>>>>>> Fourth, a million line program is too large to put into your head. You
>>>>>>> need to have some background on the data structures, control flow, any
>>>>>>> special tricks (e.g. funcalls through the symbol-plist), database
>>>>>>> design, communication protocols (e.g. between Axiom and Hyperdoc and
>>>>>>> Graphics), parsing structures, and a million other details. At best,
>>>>>>> the code tells you HOW it does something but not WHY, not what it
>>>>>>> depends on, not what depends on it, etc.
>>>>>>>
>>>>>>> Fifth, a program this large and this long-lived will eventually no
>>>>>>> longer have the authors around to ask questions. Several of the Axiom
>>>>>>> authors are already dead and most are not associated with it anymore.
>>>>>>> Some of those authors are the world's expert in their subject matter.
>>>>>>> That is a steep hill to climb if you want to understand the code,
>>>>>>> especially if you have to debug or modify it.
>>>>>>>
>>>>>>> Sixth, in the 30 year horizon view, we need to pursue a new level of
>>>>>>> excellence. As Knuth said:
>>>>>>>
>>>>>>>        "Literate programming is what you need to rise above
>>>>>>>         the ordinary level of achievement"
>>>>>>>
>>>>>>> For all these reasons, and more, Axiom needs to be literate.

\start
Date: Fri, 18 Nov 2011 14:34:48 -0500
From: Tim Daly
To: Eugene Surowitz
Subject: Re: Literate Programming -- Knuth interview

I just wrote an example of a literate program using HTML
syntax. There is a professor at Dartmouth who is willing
to talk to the class about it but found Knuth's version
too confusing. See

http://axiom-developer.org/axiom-website/litprog.html

In my opinion Knuth made the mistake of showing his mature
working environment that contains too many bells and whistles.
I have simplified the whole idea down to a simple tangle
program.

That said, I believe Knuth's idea of literate programming is
really a fundamental breakthrough. 

On Fri, 2011-11-18 at 12:53 -0500, Eugene Surowitz wrote:
> I was just at BICA2011 and they both record the talks and
> are actually creating DVDs along with the collected slides.
> (They sold last year's DVDs at this years conference.)
> Some are talks are on the web site.
> 
> I'm thinking along the line of a "text crawler" complemented
> by a relational data base with the capability to call up
> the literate pamphlets at a click.  Thus reverse walking
> gives a product support and development mechanism.
> The net result would be the ability to locate "foobar"
> whatever context it is in - code or description.

Google already does this I believe.

> I am assuming the however well Axiom gets pamphleteer-ed (ouch)
> the perversity of human nature will result in individuals
> shoehorning and shotgunning in code and comments in when
> and where it occurs to them.

The hope is that people will start making literate programming
part of their code development in groups. When you get to a code
review your code should be written as a section in a book. You
distribute this section before the code review and people can
now read "why" you wrote things.

If that happened then the code reviews would not amount to a
critique of "where to put braces" but a review of the whole
section, code and explanation.

You would not be able to check in code unless the whole thing
was clear and readable.

I would even advocate that there should be an Editor-in-chief
on the team. That is, an English major rather than a programmer.

> I would hope that the easier it is to do literate code cleanly
> would discourage that phenomena.

I started out using literate programming to try to preserve Axiom.
I think Axiom is a modern "Newton's Notebook" that arose from
the collision of math and computers. I have since learned that
literate programming makes me a much better programmer. It really
does change the quality of programs.

I've been pushing this professor to stop teaching people to
program like it is 1970 and start creating a new generation of
professional programmers who write clear, clean code that can
be understood and maintained.

The hard part is that literate programming is like lisp...
you struggle with it and struggle with it and wonder why anyone
would write code this way... and THEN AH-HA! you GET IT! And
then you wonder why anyone would write code any other way.

We can be better programmers.

> 
> Eugene J. Surowitz
> 
> On 11/14/2011 7:12 PM, daly wrote:
> > On Mon, 2011-11-14 at 14:08 -0500, Eugene Surowitz wrote:
> >> Will your talk and slides be available on the web?
> >
> > I gave a presentation at the Clojure Conj on Literate
> > Programming which was fairly well received but there were
> > no slides. I don't believe the talk was recorded.
> >
> > I have an article in the Notices of the American Mathematical
> > Society scheduled to appear in February.
> >
> >>
> >> Long term objective excellent; getting there is the rub.
> >> It's ok at the developed and distributed level,
> >> but from the developer's ant eye view the toc and index
> >> are only as good as the choice of structure and
> >> the entry selection.
> >
> > Indeed, that's a struggle for all authors. However, I have
> > made the index and cross references for every function and
> > variable.
> >
> >>
> >> There is also the headache of what the eyeball missed
> >> and what opinion bias does or does not consider important.
> >> Also texts tend to have concepts expressed as multiword
> >> strings, such as "table of contents".
> >> Pseudo-letters, such as "_" or "~", just add to the mess.
> >
> > Every function, macro, and variable is introduced with
> >       \defun{foo}
> >       \defmacro{foo}
> >       \defvar{foo}
> >
> > All of the index entries are hand generated using latex as in:
> >       \calls{foo}{bar}
> > which will put an entry under "foo" for "bar" and
> > an entry under "bar" for "foo". This means that it is trivial
> > to find every place that "bar" is called. This reverse-lookup
> > usually requires a code walker.
> >
> > For global variables there are:
> >       \refsdollar{foo}{var}
> >       \defsdollar{foo}{var}
> > which will put an entry in the index for $var.
> >
> >
> >>
> >> Exhaustive listing of everything was quite useful
> >> in another book effort.  For a code the size of Axiom,
> >> I guesstimate that a factor of three for the token file size.
> >>
> >> Eugene J. Surowitz
> >>
> >> On 11/12/2011 9:16 PM, daly wrote:
> >>> Well, the question is whether you're looking for a concept
> >>> (check table of contents) or a particular function (check
> >>> the index). At least that is the long term intention.
> >>>
> >>> On Fri, 2011-11-11 at 11:03 -0500, Eugene Surowitz wrote:
> >>>> I prefer the fountain pen for novelistic efforts.
> >>>>
> >>>> The issue that aroused my question has to do with
> >>>> comments that appear now and then in the list
> >>>> about knowing what is where in the source code.
> >>>>
> >>>> The reference to tools really was intended to address
> >>>> the idea of the utility of a mechanism(s) that could
> >>>> globally treat the entire source code as text to
> >>>> answer that question.
> >>>>
> >>>> Eugene J. Surowitz
> >>>>
> >>>> On 11/8/2011 4:42 PM, daly wrote:
> >>>>> Tools? The goal of literate programming is communicating from
> >>>>> human to human. It is like writing a novel. All you need is a
> >>>>> working Underwood typewriter and time.
> >>>>>
> >>>>> I tend to favor Latex because a lot of math is involved and
> >>>>> Latex prints math well. But I'm presenting a talk about
> >>>>> Literate Programming in North Carolina this week and I plan
> >>>>> to show examples that use XML, html, and video.
> >>>>>
> >>>>> Tim
> >>>>>
> >>>>> On Tue, 2011-11-08 at 15:03 -0500, Eugene Surowitz wrote:
> >>>>>> That literate programming is fully justified for Axiom is, well, almost axiomatic.
> >>>>>> But the issue is more how to boost the ability to invert the process
> >>>>>> and reverse engineer non-literate code piles into literate documents.
> >>>>>>
> >>>>>> What, in your opinion, would be the most effective type of tool that could
> >>>>>> be developed to render the literacy project more tractable?
> >>>>>>
> >>>>>> Eugene J. Surowitz
> >>>>>>
> >>>>>> On 11/8/2011 9:46 AM, Tim Daly wrote:
> >>>>>>>> Yet to me, literate programming is certainly the most important thing
> >>>>>>>> that came out of the TeX project. Not only has it enabled me to write
> >>>>>>>> and maintain programs faster and more reliably than ever before, and
> >>>>>>>> been one of my greatest sources of joy since the 1980s -- it has
> >>>>>>>> actually been indispensable at times. Some of my major programs, such
> >>>>>>>> as the MMIX meta-simulator, could not have been written with any other
> >>>>>>>> methodology that I've ever heard of. The complexity was simply too
> >>>>>>>> daunting for my limited brain to handle; without literate programming,
> >>>>>>>> the whole enterprise would have flopped miserably.
> >>>>>>>>
> >>>>>>>> If people discover nice ways to use the newfangled multithreaded
> >>>>>>>> machines, I would expect the discovery to come from people who
> >>>>>>>> routinely use literate programming. Literate programming is what you
> >>>>>>>> need to rise above the ordinary level of achievement.
> >>>>>>>
> >>>>>>> I believe that Axiom's complexity is large enough to demand literate
> >>>>>>> programming. There are several reasons.
> >>>>>>>
> >>>>>>> First, computational mathematics requires people to be exceptional at
> >>>>>>> mathematics and programming. This is a small subset of already small
> >>>>>>> sets. We might as well add another subset of those who can communicate
> >>>>>>> their ideas in writing.
> >>>>>>>
> >>>>>>> Second, there are many design decisions that are necessary to reduce
> >>>>>>> a mathematical idea to implementation. Some of these design decisions
> >>>>>>> are mathematically arbitrary (e.g. branch cuts) or computationally
> >>>>>>> arbitrary (e.g. sparse versus dense) or programatically arbitrary
> >>>>>>> (e.g. all Spad versus Spad-and-lisp). These design decisions need to
> >>>>>>> be documented so people who maintain and modify the program know why
> >>>>>>> the decisions were made. Without this knowledge it would be trivial
> >>>>>>> to accidently destroy important optimizations.
> >>>>>>>
> >>>>>>> Third, nobody is an expert in the range of mathematics that Axiom can
> >>>>>>> and will implement. It is important to present some portions of the
> >>>>>>> theory associated with domains so people have a clue about the ideas
> >>>>>>> being encoded. Imagine what would happen if all of the math textbooks
> >>>>>>> only contained equations but no human-readable text. You might be able
> >>>>>>> to "read" a calculus textbook but not an infinite group theory textbook.
> >>>>>>>
> >>>>>>> Fourth, a million line program is too large to put into your head. You
> >>>>>>> need to have some background on the data structures, control flow, any
> >>>>>>> special tricks (e.g. funcalls through the symbol-plist), database
> >>>>>>> design, communication protocols (e.g. between Axiom and Hyperdoc and
> >>>>>>> Graphics), parsing structures, and a million other details. At best,
> >>>>>>> the code tells you HOW it does something but not WHY, not what it
> >>>>>>> depends on, not what depends on it, etc.
> >>>>>>>
> >>>>>>> Fifth, a program this large and this long-lived will eventually no
> >>>>>>> longer have the authors around to ask questions. Several of the Axiom
> >>>>>>> authors are already dead and most are not associated with it anymore.
> >>>>>>> Some of those authors are the world's expert in their subject matter.
> >>>>>>> That is a steep hill to climb if you want to understand the code,
> >>>>>>> especially if you have to debug or modify it.
> >>>>>>>
> >>>>>>> Sixth, in the 30 year horizon view, we need to pursue a new level of
> >>>>>>> excellence. As Knuth said:
> >>>>>>>
> >>>>>>>        "Literate programming is what you need to rise above
> >>>>>>>         the ordinary level of achievement"
> >>>>>>>
> >>>>>>> For all these reasons, and more, Axiom needs to be literate.

\start
Date: Fri, 18 Nov 2011 20:24:33 -0800 (PST)
From: Cliff Yapp
To: Tim Daly
Subject: Re: Literate Programming -- Knuth interview

Tim, That's quite an interesting example! Is there a license on it? I
can see that being useful in a lot of scenarios as an introduction to
the idea of literate programming.

\start
Date: Sat, 19 Nov 2011 00:03:35 -0500
From: Tim Daly
To: Cliff Yapp
Subject: Re: Literate Programming -- Knuth interview

On Fri, 2011-11-18 at 20:24 -0800, C Y wrote: 
> Tim, 
> 
> That's quite an interesting example!  Is there a license on it?  I can
> see that being useful in a lot of scenarios as an introduction to the
> idea of literate programming.

License? Nope.

You have my permission to use it in any fashion anywhere,
at any time, for any purpose.

It would be fun to create a viral-literate license that
required you to write all your published programs that way :-)

I posted the example to the Clojure mailing list and got 
a response which I include here.

On Fri, 2011-11-18 at 20:02 -0800, Daniel Jomphe wrote:
> On Friday, November 18, 2011 7:17:08 AM UTC-5, TimDaly wrote:
>         Many of you asked me to show an example of a literate
>         program and demonstrate the use of the tangle function.
>         
>         I usually use Latex for literate work but I've done
>         this example using HTML and <pre id="foo"> tags.
>         
>         I've written a self-referential literate program that
>         explains the details of the tangle function in literate
>         form. You can find the web page at
>         
>         http://daly.literatesoftware.com/lithtml/litprog.html
>         
> I have read your literate program, and must recognize that I know how
> tangle works even though I didn't want to really read the source code.
> I read your prose all the way through. I still haven't read the source
> code; 

In fact, that's the whole point. You don't read the equations in a
calculus textbook either. You read the words. The equations are icons.
If you understood the text and "spoke" mathematics you could probably
write the equations.

In programming we can reach the same level of literacy. Reading just
the words in the literate version it should be possible to recreate
the program in your favorite language. Note that you would be creating
a different program with different design decisions but the same 
functionality. 

> I didn't feel the need to read it. Were I to maintain your program,
> I'd have more than enough confidence to start hacking the code right
> now.

One thing worth trying would be to code the same program in Clojure.

The tangle program is conceptually very simple but there are a lot
of low level design decisions that I would make differently. For
example, there are loops in the C program which would go away.

Would you map read or would you slurp? Mapping a read function allows
transforming "& lt;" to < at read time. This does not matter in the C
program because the buffer is mutable but it would matter in Clojure.

Would you use the Clojure pattern language to find the <pre> tags?
Would you be able to parse out the string from the id? C encourages
character-level hacking but Clojure would be much more powerful.

> 
> 
> I think this speaks very positively about literate programming. What
> remains to be seen is how much (or not) I'm going to practice it in
> the future.

If you do try to rewrite it in Clojure please post the program. I 
would be very interested to see how Clojure's concise syntax and
semantics get reflected in your design decisions.

The tangle program in Clojure might turn out to be a single
s-expression of only a few lines. The code density would be a
huge win but a literate version would still have to have the
vitals of the story. Remember that the key test for a literate
program is the "independence test". Someone can read it without
talking to you, understand how it works, and be able to change it.

> 
> 
> What do you think of marginalia? It's a bit the reverse of tangle; it
> assembles all those 70's files together into this book you might want
> to read. Is it sound or not? Have your thoughts changed from what you
> wrote in [1]?
> 
> 
> [1] http://goo.gl/cXWzF

Literate programming is a mindset, not a tool. You can write a 
literate program in anything, including marginalia.

That said, I have yet to see a Clojure program that lays out a story
so I can sit and read it. For a real challenge, if you try to write
tangle in Clojure, try writing the story in marginalia. I'm sure
Fogus would welcome the feedback.


The readability aspect is a real feature. Heck, you could even give 
your programs to a company so they could read them BEFORE the job
interview. I would strongly favor hiring someone who could 
communicate, who cared about code quality, and who could improve
the company's maintenance headache in the long term. A Literate
Clojure programmer would be a real gotta-hire person. Companies
use many programming languages but all programmers really do need
good communication skills. 


In the long view, it would be sweet if the Clojure reader knew how
to read a literate program. Just call "(literate-load file chunk)" 
and you get the same effect as if you had tangled the program to 
a file.

With literate-load available you would be able to write all of
your Clojure code in a literate style, making Clojure much 
easier to understand, maintain, and modify.

\start
Date: Sat, 19 Nov 2011 08:42:42 +0000
From: Martin Baker
To: list
Subject: Re: Literate Programming -- Knuth interview

One issue that occurred to me on this subject. It seems to me the point of 
HTML and hypertext is that they are not a linear book. What I like about this 
is that the reader can start at a high level with a small and concise page but 
they can drill down to any level of detail they may need. Also there can be 
different entry points for different types of readers such as end users as 
well as programmers. It would be good if all these types of 'documentation' 
could be produced by the same tools and advanced end users could drill down to 
information originally intended for programmers. Using standard code libraries 
its probably easier to access HTML user documentation/'help files' from within 
the program rather than writing custom code like hyperdoc.

I can see the advantages in putting the literate 'documentation' interleaved 
in the same file as the code. However there also seem to be a lot of 
disadvantages such as making the above ideas more difficult?

\start
Date: Sat, 19 Nov 2011 04:30:57 -0500
From: Tim Daly
To: Martin Baker
Subject: Re: Literate Programming -- Knuth interview

On Sat, 2011-11-19 at 08:42 +0000, Martin Baker wrote:
> Tim,
 
> One issue that occurred to me on this subject. It seems to me the
> point of HTML and hypertext is that they are not a linear book. What
> I like about this is that the reader can start at a high level with
> a small and concise page but they can drill down to any level of
> detail they may need. Also there can be different entry points for
> different types of readers such as end users as well as
> programmers. It would be good if all these types of 'documentation'
> could be produced by the same tools and advanced end users could
> drill down to information originally intended for programmers. Using
> standard code libraries its probably easier to access HTML user
> documentation/'help files' from within the program rather than
> writing custom code like hyperdoc.

I chose HTML as the example case because it is known to everyone.
I much prefer latex but not everyone does.

Axiom has an implementation of an HTML replacement for hyperdoc in
Volume 11. There is still much work to be done.

I believe the split between end users and programmers is artificial
and an artifact of the way we currently develop code.

Consider writing a textbook. You start out writing what you believe will
be the first chapter, diving right into the interesting material. 
Eventually you find that Chapter 1 requires some background material
so you write a new Chapter 1 and your interesting material is now in
Chapter 2. As you continue this process you find that the material
you originally put in Chapter 1 is now somewhere near Chapter 4 because
you found many ideas that needed to be explained before you could really
cover the interesting material. 

Often I will read the first 3 chapters of a textbook, then find another
textbook on the same subject and read those first 3 chapters, and even
a third textbook. By the time I've done that I have a firm grasp on the
background essentials and find that I can make progress on later 
chapters with core material.

We have this belief that programmers have a natural ability to dive
into code that is lacking in other people. We assume that someone who
"speaks" C can just read the program and understand it. In my 
experience this is a bad assumption. Programmers have no way to
guess at a design decision that is not written in words any better
than anyone else. Nor do programmers have a magic mind that allows 
them to look at several hundred "sand files" and derive the big 
ideas of the program. Yet we continue to act as if they do.

My experience with literate programming has led me to expect that I
have to write the "first three chapters" (sections, paragraphs, or
whatever) as general overview. I have to make gradual steps from
ideas to implementation. Done properly I believe anyone can read a
literate program, at least superficially, and especially so for the
first few chapters.

After all, the point of literate programming is to communicate ideas
from one human to another. 

The organization of the material into one or many web pages is not
particularly important. I tend to follow the "book" metaphor as
I tend to read a lot of books. Linear material spread over a lot of
pages simply breaks my "linear mind" and I tend to stop reading after
getting a few pages into the material. However, with a PDF style
book my Kindle or PDF reader keeps my last-read-page bookmark and
I can pick up where I left off. 


> 
> I can see the advantages in putting the literate 'documentation' interleaved 
> in the same file as the code. However there also seem to be a lot of 
> disadvantages such as making the above ideas more difficult?

The key test for literate documentation is the "independence test".
Can you point someone at your literate program, have them go off
somewhere to read it, and then come back with the ability to maintain
and modify it?

When you read the tangle web page did you feel that it needed to be
broken up? I thought it made a reasonably smooth transition from the
ideas to the code details with enough words so you could skip reading
the actual code. The code is just there as icons for humans, like
equations in a math book.

If you feel that interleaving videos, animations, navigation, and any
other tricks improves the "independence test" results then these
techniques should be used. If, however, you are just trying to 
organize the material by some random criteria (e.g. alphabetical
or as trees of logically grouped files) then you are undermining 
the transfer of ideas.

Even today, with all of the hyperlinking tricks available, I still
see my iPad using linear, book-like presentations.

\start
Date: Sat, 19 Nov 2011 10:36:46 +0000
From: Martin Baker
To: list
Subject: Re: Literate Programming -- Knuth interview

> If you feel that interleaving videos, animations, navigation, and any
> other tricks improves the "independence test" results then these
> techniques should be used. If, however, you are just trying to
> organize the material by some random criteria (e.g. alphabetical
> or as trees of logically grouped files) then you are undermining
> the transfer of ideas.

Tim,

I think we may agree about the importance and general principles of this and 
we are just discussing the mechanics here? (although I think that navigation 
and diagrams are much more fundamental and powerful than just "tricks")

I think its good to take into account practical matters and allow the use of 
standard HTML and code editors and to separate these principles from the tools 
used.

I don't see problem in having multiple source, HTML and diagram files treated 
as single entity (in a ZIP file or something similar) the entry points would 
be at the root. I cant see a problem for the "independence test".

When the code is complied the HTML files can be transferred unchanged to the 
runtime so they are available for hyperdoc(HTML version) and help files at 
runtime.

\start
Date: Sat, 19 Nov 2011 05:41:48 -0500
From: Tim Daly
To: Martin Baker
Subject: Re: Literate Programming -- Knuth interview

On Sat, 2011-11-19 at 10:17 +0000, Martin Baker wrote:
> > If you feel that interleaving videos, animations, navigation, and any
> > other tricks improves the "independence test" results then these
> > techniques should be used. If, however, you are just trying to
> > organize the material by some random criteria (e.g. alphabetical
> > or as trees of logically grouped files) then you are undermining
> > the transfer of ideas.
> 
> Tim,
> 
> I think we may agree about the importance and general principles of this and 
> we are just discussing the mechanics here? (although I think that navigation 
> and diagrams are much more fundamental and powerful than just "tricks")

I think we are in agreement on principles I still shy away from a 
focus on the mechanics. Think typewriters.

>  I think its good to take into account practical matters and allow
> the use of standard HTML and code editors and to separate these
> principles from the tools used.

Well, maybe. Programmers love their tools and see nothing wrong with 
splashing "documentation" all over a tree of little files. You could
write a literate program that way but it would be like reading a
book in weekly installments in a magazine.

> 
> I don't see problem in having multiple source, HTML and diagram files treated 
> as single entity (in a ZIP file or something similar) the entry points would 
> be at the root. I cant see a problem for the "independence test".

Several people have pointed me at trees of code that are supposed to be
in a literate style. The problem is that they have confused the idea of
documentation or comments with the idea of literate programming.

Literate programming makes the human to human communication paramount.
This means that you ought to be able to "sit and read", like any good
novel. You could easily print the example page I used and read it from
beginning to end without a computer.

Documentation and comment systems are not like this. They make the
program organization "fit the machine". They talk about the code, and
focus on line-by-line or file-by-file. They tend to work well with all
kinds of "tools" like Eclipse, Javadoc, or Doxygen.

It is easy to confuse literate programming with these documentation
and commenting system but they are nowhere the same.

So if I have to know anything about file organization, if I have to
pop between files or if I have to have rules about what the correct
syntax is (e.g. Javadoc) then I am looking at a documentation and
comment system.

If I can write a story in any form I want and splash code icons in at
the appropriate moment, if the story focuses on ideas, and if I can
understand it without looking at the code then I am looking at a
literate program.

> 
> When the code is complied the HTML files can be transferred unchanged to the 
> runtime so they are available for hyperdoc(HTML version) and help files at 
> runtime.

That's a documentation and commenting tool.

\start
Date: Sat, 19 Nov 2011 16:11:30 +0000
From: Martin Baker
To: list
Subject: Re: Literate Programming -- Knuth interview

> Documentation and comment systems are not like this. They make the
> program organization "fit the machine". They talk about the code, and
> focus on line-by-line or file-by-file. They tend to work well with all
> kinds of "tools" like Eclipse, Javadoc, or Doxygen.

I tend think of this as reference information. The sort of thing that Axiom 
produces dynamically at runtime like )show and )display. I think there would 
be advantages in generating this statically with a compile-time documentation 
tool, the information would be much more richly interlinked with itself and 
the other type of information.

> It is easy to confuse literate programming with these documentation
> and commenting system but they are nowhere the same.

They may not be the same but why should the user have to switch between them 
and know which to use and when?

In Axiom, I have often tried to find something in pdf generated by literate 
code when what I wanted was in hyperdoc or runtime commands or visa-versa. 
There seems to be a lot of overlap of these sources of information in Axiom. I 
would like to be able to link between them by clicking on the text.

Also I may be reading something as a story when I come across some issue that 
I need to clarify which may be a different type of documentation and so I need 
to switch to a more reference way of working. I think there are all kinds of 
things that the user/maintainer needs to do between these extremes and I find 
well produced HTML to be the best way to get to the information that I need 
(of course, like anything else, there is a lot of bad HTML around).

Also I think a tree structure scales up better than a linear structure.  A 
story works well for a small program, but for a big program like Axiom, there 
has to be some form of structure doesn't there? 

\start
Date: Sat, 19 Nov 2011 14:09:47 -0500
From: Tim Daly
To: Martin Baker
Subject: Re: Literate Programming -- Knuth interview

On Sat, 2011-11-19 at 16:11 +0000, Martin Baker wrote:
> > Documentation and comment systems are not like this. They make the
> > program organization "fit the machine". They talk about the code, and
> > focus on line-by-line or file-by-file. They tend to work well with all
> > kinds of "tools" like Eclipse, Javadoc, or Doxygen.
> 
> I tend think of this as reference information. The sort of thing that Axiom 
> produces dynamically at runtime like )show and )display. I think there would 
> be advantages in generating this statically with a compile-time documentation 
> tool, the information would be much more richly interlinked with itself and 
> the other type of information.
> 
> > It is easy to confuse literate programming with these documentation
> > and commenting system but they are nowhere the same.
> 
> They may not be the same but why should the user have to switch between them 
> and know which to use and when?
> 
> In Axiom, I have often tried to find something in pdf generated by literate 
> code when what I wanted was in hyperdoc or runtime commands or visa-versa. 
> There seems to be a lot of overlap of these sources of information in Axiom. I 
> would like to be able to link between them by clicking on the text.
> 
> Also I may be reading something as a story when I come across some issue that 
> I need to clarify which may be a different type of documentation and so I need 
> to switch to a more reference way of working. I think there are all kinds of 
> things that the user/maintainer needs to do between these extremes and I find 
> well produced HTML to be the best way to get to the information that I need 
> (of course, like anything else, there is a lot of bad HTML around).
> 
> Also I think a tree structure scales up better than a linear structure.  A 
> story works well for a small program, but for a big program like Axiom, there 
> has to be some form of structure doesn't there? 

Frankly, I don't know. I am learning what works and what doesn't as I
go. Axiom is the largest literate program I'm aware of so there are no
examples. I get to invent the mistakes.

Volume 12 is about the Crystal. The crystal idea is that there is a core
problem. Wrapped around the problem is a crystal with many facets. Each
facet presents a different view of the core problem. My thinking is
that one facet presents a PDF view of related material, say by showing
a particular section of the book. Another facet presents related
hyperdoc material. A third facet presents related code. A fourth
facet presents related material in HTML form, with embedded video
or animations. A fifth facet would show a graph, like the domain
graph, related to the problem. You "rotate" the crystal to find the
facet that has the information you want.

So I think there is a place for all of the various representations.
They need to be driven from the core problem. To my knowledge, there
are no examples and no effort anywhere to develop a crystal and 
facets idea. So this is all just one big research experiment on 
my part.

\start
Date: Sat, 19 Nov 2011 14:37:48 -0500
From: Tim Daly
To: clojure@googlegroups.com
Subject: Re: Literate Programming example

On Sat, 2011-11-19 at 14:35 +0100, Laurent PETIT wrote:
> Hello,
> 
> 2011/11/19 Tim Daly
>         On Fri, 2011-11-18 at 20:02 -0800, Daniel Jomphe wrote:
>         > On Friday, November 18, 2011 7:17:08 AM UTC-5, TimDaly
>         wrote:
>         >         Many of you asked me to show an example of a
>         literate
>         >         program and demonstrate the use of the tangle
>         function.
>         >
>         >         I usually use Latex for literate work but I've done
>         >         this example using HTML and <pre id="foo"> tags.
>         >
>         >         I've written a self-referential literate program
>         that
>         >         explains the details of the tangle function in
>         literate
>         >         form. You can find the web page at
>         >
>         >
>         http://daly.literatesoftware.com/lithtml/litprog.html
>         >
>         > I have read your literate program, and must recognize that I
>         know how
>         > tangle works even though I didn't want to really read the
>         source code.
>         > I read your prose all the way through. I still haven't read
>         the source
>         > code;
>         
>         
>         In fact, that's the whole point. You don't read the equations
>         in a
>         calculus textbook either. You read the words. The equations
>         are icons.
>         If you understood the text and "spoke" mathematics you could
>         probably
>         write the equations.
>         
>         In programming we can reach the same level of literacy.
>         Reading just
>         the words in the literate version it should be possible to
>         recreate
>         the program in your favorite language. Note that you would be
>         creating
>         a different program with different design decisions but the
>         same
>         functionality.
>         
>         > I didn't feel the need to read it. Were I to maintain your
>         program,
>         > I'd have more than enough confidence to start hacking the
>         code right
>         > now.
>         
>         
>         One thing worth trying would be to code the same program in
>         Clojure.
>         
>         The tangle program is conceptually very simple but there are a
>         lot
>         of low level design decisions that I would make differently.
>         For
>         example, there are loops in the C program which would go away.
>         
>         Would you map read or would you slurp? Mapping a read function
>         allows
>         transforming "& lt;" to < at read time. This does not matter
>         in the C
>         program because the buffer is mutable but it would matter in
>         Clojure.
>         
>         Would you use the Clojure pattern language to find the <pre>
>         tags?
>         Would you be able to parse out the string from the id? C
>         encourages
>         character-level hacking but Clojure would be much more
>         powerful.
>         
>         >
>         >
>         > I think this speaks very positively about literate
>         programming. What
>         > remains to be seen is how much (or not) I'm going to
>         practice it in
>         > the future.
>         
>         
>         If you do try to rewrite it in Clojure please post the
>         program. I
>         would be very interested to see how Clojure's concise syntax
>         and
>         semantics get reflected in your design decisions.
>         
>         The tangle program in Clojure might turn out to be a single
>         s-expression of only a few lines. The code density would be a
>         huge win but a literate version would still have to have the
>         vitals of the story. Remember that the key test for a literate
>         program is the "independence test". Someone can read it
>         without
>         talking to you, understand how it works, and be able to change
>         it.
>         
>         >
>         >
>         > What do you think of marginalia? It's a bit the reverse of
>         tangle; it
>         > assembles all those 70's files together into this book you
>         might want
>         > to read. Is it sound or not? Have your thoughts changed from
>         what you
>         > wrote in [1]?
>         >
>         >
>         > [1] http://goo.gl/cXWzF
>         
>         
>         Literate programming is a mindset, not a tool. You can write a
>         literate program in anything, including marginalia.
>         
>         That said, I have yet to see a Clojure program that lays out a
>         story
>         so I can sit and read it. For a real challenge, if you try to
>         write
>         tangle in Clojure, try writing the story in marginalia. I'm
>         sure
>         Fogus would welcome the feedback.
>         
>         
>         The readability aspect is a real feature. Heck, you could even
>         give
>         your programs to a company so they could read them BEFORE the
>         job
>         interview. I would strongly favor hiring someone who could
>         communicate, who cared about code quality, and who could
>         improve
>         the company's maintenance headache in the long term. A
>         Literate
>         Clojure programmer would be a real gotta-hire person.
>         Companies
>         use many programming languages but all programmers really do
>         need
>         good communication skills.
>         
>         
>         In the long view, it would be sweet if the Clojure reader knew
>         how
>         to read a literate program. Just call "(literate-load file
>         chunk)"
>         and you get the same effect as if you had tangled the program
>         to
>         a file.
>         
>         With literate-load available you would be able to write all of
>         your Clojure code in a literate style, making Clojure much
>         easier to understand, maintain, and modify.
> 
> 
> I'd be interested in studies proving that programs written in a
> litterate style are easier to maintain.
> 
> 
> So far, there has been this metaphore between a "story" and a
> "program". How far can this metaphor be pushed ?
> 
> 
> Is it easy to write a book with more than a couple people ? To
> maintain it ? To modify it ?
> 
> 
> I can easily understand the appeal: if I were to discover a program
> for the first time, I would certainly be interested in discovering it
> via a literate style, because I would have some guarantee that some
> effort has been pushed into providing for the mainteners: contexts,
> argumentations, debates concerning design choices, etc.
> 
> 
> But past this first "introduction" to the program, I don't know what
> it would cost to maintain it without adding to the entropy of its
> literate part.
> 
> 
> I mean: to be honest, it's not as if, already, all programs written
> were correctly documented and we were facing a wall: "we can not
> improve our documentation further, we need to find another way to
> document our programs" ;-).
> 
> 
> Sorry to have to play the role of the devil's advocate, but I think
> there are currently holes in the argumentation in favor of literate
> programming, at least wrt to the material I've been exposed to so
> far ...
> 

YOU signed a contract with the Devil too? Hmmm. I thought I had an
exclusive contract. Perhaps he lied to me. :-)

I have experience what most programmers never see. I got my own code
back after many years because I volunteered to open source the Axiom
code. I was one of the original authors. 

I write dirt simple lisp code. I know exactly what it does. However,
I don't know why I wrote it. I don't know how it fits into the whole.
I was clearly solving a problem but the problem definition is gone
even though the code is there. I know I need the code because the
system crashes if I delete it.

I'm left with the problem of reverse-engineering the code. All I
can do is invent probable reasons. Nobody wrote down the design
decisions. It is very difficult to maintain and modify 1 million
lines of lisp code.

If you want to see the difference a literate program would make I
suggest taking one of Rich's ideas, such as software transactional
memory, and try to explain the code that implements it. I have
tried that in my Clojure in Small Pieces book and it is a LOT
harder than it looks.

Literate programming is not for you. In fact, it is probably not
worthwhile for most programs. But if you're going to make a program
that "lives" beyond you then it really matters. You need to communicate
to people you will never talk to and who cannot ask you questions.
Some of the authors of Axiom are already dead. They don't seem to
be answering my emails.

As a lifelong lisper I've decided that Clojure is interesting to me.
I'm willing to devote a lot of time and attention to it. In particular,
I want it to outlive Rich Hickey's involvement in it. Rich will
eventually retire into management somewhere and the code will begin
to rot. I am hoping that a literate form of Clojure will make it
possible for new people to maintain, modify, and extend it.

To that end, I'd like a literate form of Clojure that passes the
"independence test". That is, I'd like a new person to be able to
take the literate Clojure Core off to an island for a couple weeks
of reading and come back as a well informed, productive developer
capable of contributing correctly without breaking existing ideas.

Thus, literate programming is an investment in the future.

However, as Knuth points out and as I've already experienced, writing
a program in literate form vastly reduces the errors. There are two
causes I can find. 

First, if I have to write an explanation then I have to justify my
poor code hacks. Sometimes I find that I rewrite the code because the
very act of explaining the bad code made me realize that the code is
bad. I find I have to talk about memory allocations so I catch leaks.
I have to talk about argument handling so I end up proving that the
arguments are correct, or I end up checking for possible failures.

Second, because other people can see the reasoning at the code review,
they can chastise me for failing to explain, or explaining things that
the code does not do, or just plain failing to implement a correct
solution. 

I have found that literate programming makes me a better programmer.

Convincing people to use literate programming is like convincing
them to use lisp. Before the AH-HA moment occurs they can't imagine
why anyone would code this way. After the AH-HA moment they can't
imagine why anyone would code any other way. I have clearly crossed
that line.

\start
Date: Sat, 19 Nov 2011 13:52:35 -0600
From: Raymond Rogers
To: Tim Daly
Subject: Re: Literate Programming example

It's been a long time but I believe I read
Anatomy of Lisp
John Allen

As I recall it would be an example of "literate programing" in the
sense that every piece of code had extensive explanation before and
after; and could actually lift the code and implement it.  I found it
a revelation; he actually built a lisp interpreter and a memory
management DB system right in front of my eyes; and I understood it.
I had always thought that starting at that level and generating what
he did wouldn't be possible without inventing a new language for the
DB handling.  I thought DB handling was totally inappropriate for
Lisp.  Of course he did start from a (small) kernel.  Sad to say it's
out of print but Amazon has it available; for a exorbitant price IMHO.

\start
Date: Sat, 19 Nov 2011 15:07:02 -0500
From: Tim Daly
To: Raymond Rogers
Subject: Re: Literate Programming example

I loved that book. I taught from it at Vassar.
It is well worth the price.

On Sat, 2011-11-19 at 13:52 -0600, Raymond Rogers wrote:
> It's been a long time but I believe I read
> Anatomy of Lisp
> John Allen
> 
> As I recall it would be an example of "literate programing" in the sense
> that every piece of code had extensive explanation before and after; and
> could actually lift the code and implement it.
> I found it a revelation; he actually built a lisp interpreter and a
> memory management DB system right in front of my eyes; and I understood
> it.  I had always thought that starting at that level and generating
> what he did wouldn't be possible without inventing a new language for
> the DB handling.  I thought DB handling was totally inappropriate for Lisp.
> Of course he did start from a (small) kernel.
> Sad to say it's out of print but Amazon has it available; for a
> exorbitant price IMHO.

\start
Date: Sat, 19 Nov 2011 13:21:50 -0800 (PST)
From: Daniel Jomphe
To: clojure@googlegroups.com
Subject: Re: Literate Programming example

On Saturday, November 19, 2011 2:37:48 PM UTC-5, TimDaly wrote:
>
> However, as Knuth points out and as I've already experienced, writing
> a program in literate form vastly reduces the errors. There are two
> causes I can find. 
>
> First, if I have to write an explanation then I have to justify my
> poor code hacks. Sometimes I find that I rewrite the code because the
> very act of explaining the bad code made me realize that the code is
> bad. I find I have to talk about memory allocations so I catch leaks.
> I have to talk about argument handling so I end up proving that the
> arguments are correct, or I end up checking for possible failures.
>
> Second, because other people can see the reasoning at the code review,
> they can chastise me for failing to explain, or explaining things that
> the code does not do, or just plain failing to implement a correct
> solution.
>
With the tools available to us today, there's no reason why we at least 
shouldn't have everything needed to make literate programming more 
seamless, more natural. For example, while reading your toy example, I 
found myself wanting to ask a question or comment on your thoughts a few 
times. If your book had been displayed on a dynamic website geared towards 
literate programming, I might have been able to click on a paragraph and 
write my question/comment right there. And then, after a short conversation 
there, you would have integrated the fruits of our conversation directly 
into the end result. Thus each new reader would have been an occasion to 
improve the book. ...It's nothing surprising since this kind of review 
system already exists in some publishers' toolkits.

Another thing that appeals to me regarding Tangle, is the fact that it may 
be used to keep things DRY. For example, when we have many arity-overloaded 
implementations of a function, some of their arguments are repeated, and we 
need to duplicate their api docs. Tangle could be leveraged to get rid of 
this duplication.

\start
Date: Sun, 20 Nov 2011 14:20:08 -0800 (PST)
From: Daniel Jomphe
To: clojure@googlegroups.com
Subject: Re: Literate Programming example

On Sunday, November 20, 2011 6:17:13 AM UTC-5, robermann79 wrote:
>
> FYI: some time ago the Opensuse project used such a collaborative tool
> (http://www.co-ment.com) in order to get a shared mindset of its
> goals.
> This was the result, see how clicking on higlight words points to
> their comments:
> https://lite.co-ment.com/text/lNPCgzeGHdV/view/


Interesting. And funny, considering I bought the cochapter.com domain to do 
something similar. I'm yet to deliver the application that's planned to 
power that domain.

\start
Date: Sun, 20 Nov 2011 12:17:13 +0100
From: Roberto Mannai
To: list
Subject: Re: Literate Programming example

FYI: some time ago the Opensuse project used such a collaborative tool
(http://www.co-ment.com) in order to get a shared mindset of its
goals.  This was the result, see how clicking on higlight words points
to their comments: https://lite.co-ment.com/text/lNPCgzeGHdV/view/

On Sat, Nov 19, 2011 at 10:21 PM, Daniel Jomphe wrote:
> With the tools available to us today, there's no reason why we at least
> shouldn't have everything needed to make literate programming more seamless,
> more natural. For example, while reading your toy example, I found myself
> wanting to ask a question or comment on your thoughts a few times. If your
> book had been displayed on a dynamic website geared towards literate
> programming, I might have been able to click on a paragraph and write my
> question/comment right there. And then, after a short conversation there,
> you would have integrated the fruits of our conversation directly into the
> end result. Thus each new reader would have been an occasion to improve the
> book. ...It's nothing surprising since this kind of review system already
> exists in some publishers' toolkits.

\start
Date: Wed, 30 Nov 2011 22:52:51 -0600
From: Tim Daly
To: list
Subject: Axiom release for November 2011

November 2011 Release

Treeshaking the interpreter and compiler continues and a few more files
were merged and removed.

Several books and Makefiles were converted to use lisp tangle rather 
than noweb. This is part of the general process to remove noweb.

Global function rewrites continue, including removing GETREFV and makeprop.

Work was done to improve OpenMath support. The OpenMath semantics CDs were
added and the OpenMath parser was enhanced.

More effort was put into literate work. In particular, an example of a
literate program using HTML was created and presented at the Clojure/Conj
conference. (See http://axiom-developer.org/axiom-website/litprog.html )

books/Makefile
   removed noweb, move to lisp tangle
   fix missing index files on PDfs

books/tangle.lisp 
   fix .input file algebra extraction

books/bookvol10.2
   missing/unused function cleanup
   remove makeprop
   remove noweb, move to lisp tangle

books/bookvol10.3 
   remove GETREFV
   remove makeprop
   remove noweb, move to lisp tangle

books/bookvol10.4 
   remove noweb, move to lisp tangle

books/bookvol10.5 
   remove noweb, move to lisp tangle

books/bookvol4
   missing/unused function cleanup
   remove makeprop

books/bookvol5 
   OpenMath support
   add OpenMath stubs
   missing/unused function cleanup
   remove makeprop
   remove noweb, move to lisp tangle
   treeshake interpreter

books/bookvol7.1 
   fix documentation
   update What's New Page

books/bookvol9
   missing/unused function cleanup
   remove makeprop
   treeshake and merge c-doc.lisp

books/bookvolbib
   Axiom OpenMath references
   add additional references
   add references

books/ps/
   v71releasenotes.eps add release notes
   v71sept2011.eps added

src/algebra/Makefile 
   fix .input file algebra extraction
   remove noweb, move to lisp tangle

src/axiom-website/documentation.html
   add Alexander quote
   add Knuth quote
   add litprog.html
   add quote

src/axiom-website/download.html
   update to latest release

src/axiom-website/litprog.html
   added
   add quote
   literate program example update

src/interp/Makefile 
   remove apply.lisp
   remove c-doc.lisp
   remove noweb, move to lisp tangle
   remove nruntime.lisp

src/interp
   apply.lisp removed, merged with bookvol9
   buildom.lisp remove GETREFV
   c-doc.lisp removed
   c-util.lisp missing/unused function cleanup
   category.lisp remove GETREFV
   functor.lisp missing/unused function cleanup, remove GETREFV
   g-util.lisp missing/unused function cleanup
   i-eval.lisp remove nruntime.lisp
   i-spec1.lisp treeshake interpreter
   i-spec2.lisp fix AN has sqrt: % -> %
   i-spec2.lisp remove makeprop
   nruncomp.lisp remove GETREFV, treeshake compiler
   nrungo.lisp remove nruntime.lisp, treeshake interpreter, remove GETREFV
   nruntime.lisp removed
   parsing.lisp remove makeprop
   sys-pkg.lisp missing/unused function cleanup, remove GETREFV, makeprop
   template.lisp remove GETREFV
   vmlisp.lisp missing/unused function cleanup, remove GETREFV, makeprop

zips
   add cds.tar.gz OpenMath support. add OpenMath CDs


