TITLE: Doconce Description
AUTHOR: Hans Petter Langtangen at Simula Research Laboratory and University of Oslo
DATE: today


# lines beginning with # are comment lines


======= What Is Doconce? =======
label{what:is:doconce}
idx{doconce!short explanation}

# #include "../tutorial/_what_is.do.txt"

# #include "_install.do.txt"

#
# Here are some comment lines that do not affect any formatting
# these lines are converted to comments in the output format.
# This may have some side effects, especially in rst and sphinx
# where lines following the comment may be taken as part of
# the comment if there are no blank lines after the comment.
#
# One can use ## and the mako preprocessor to remove comments
# *before* doconce sees the text. That can be useful when
# doconce comments interferes with formatting.
# The mako tool also supports <%doc> .. </%doc>
#


===== Demos =====

idx{demos}

The current text is generated from a Doconce format stored in the
!bc sys
docs/manual/manual.do.txt
!ec
file in the Doconce source code tree. We have made a
"demo web page": "https://doconce.googlecode.com/hg/doc/demos/manual/index.html"
where you can compare the Doconce source with the output in many
different formats: HTML, LaTeX, plain text, etc.

The file `make.sh` in the same directory as the `manual.do.txt` file
(the current text) shows how to run `doconce format` on the
Doconce file to obtain documents in various formats.

Another demo is found in
!bc sys
docs/tutorial/tutorial.do.txt
!ec
In the `tutorial` directory there is also a `make.sh` file producing a
lot of formats, with a corresponding
"web demo": "https://doconce.googlecode.com/hg/doc/demos/tutorial/index.html"
of the results.

# Example on including another Doconce file:

# #include "../tutorial/_doconce2anything.do.txt"



======= The Doconce Markup Language =======

The Doconce format introduces four constructs to markup text:
lists, special lines, inline tags, and environments.

===== Lists =====

An unordered bullet list makes use of the `*` as bullet sign
and is indented as follows

!bc
   * item 1

   * item 2

     * subitem 1, if there are more
       lines, each line must
       be intended as shown here

     * subitem 2,
       also spans two lines

   * item 3
!ec

This list gets typeset as

   * item 1

   * item 2

     * subitem 1, if there are more
       lines, each line must
       be intended as shown here

     * subitem 2,
       also spans two lines

   * item 3

# #if FORMAT == "gwiki"
(As seen, nested lists in (g)wiki format are not treated well by
Doconce. Plain unnested lists work fine. And the (g)wiki format
automatically puts multiple lines of an item on a single line as
required :-)
# #endif

In an ordered list, each item starts with an `o` (as the first letter
in "ordered"):

!bc
   o item 1

   o item 2

     * subitem 1

     * subitem 2

   o item 3
!ec

resulting in

   o item 1

   o item 2

     * subitem 1

     * subitem 2

   o item 3

# #if FORMAT == "gwiki"
(Again, there are problems with mixing nested lists and liststypes
for the (g)wiki format.)
# #endif

Ordered lists cannot have an ordered sublist, i.e., the ordering
applies to the outer list only.

In a description list, each item is recognized by a dash followed
by a keyword followed by a colon:

!bc
   - keyword1: explanation of keyword1

   - keyword2: explanation
     of keyword2 (remember to indent properly
     if there are multiple
     lines)
!ec

The result becomes

   - keyword1: explanation of keyword1

   - keyword2: explanation
     of keyword2 (remember to indent properly
     if there are multiple
     lines)


===== Special Lines =====

The Doconce markup language has a concept called *special lines*.
Such lines starts with a markup at the very beginning of the
line and are used to mark document title, authors, date,
sections, subsections, paragraphs., figures, movies, etc.

idx{`TITLE` keyword} idx{`AUTHOR` keyword} idx{`DATE` keyword}

__Heading with Title and Author(s).__
Lines starting with `TITLE:`, `AUTHOR:`, and `DATE:` are optional and used
to identify a title of the document, the authors, and the date. The
title is treated as the rest of the line, so is the date, but the
author text consists of the name and associated institution(s) with
the syntax
!bc
name at institution1 and institution2 and institution3
!ec
The `at` with surrounding spaces
is essential for adding information about institution(s)
to the author name, and the `and` with surrounding spaces is
essential as delimiter between different institutions.
An email address can optionally be included, using the syntax
!bc
name Email: somename@site.net at institution1 and institution2
!ec
Multiple authors require multiple `AUTHOR:` lines. All information
associated with `TITLE:` and `AUTHOR:` keywords must appear on a single
line.  Here is an example:
!bc
TITLE: On an Ultimate Markup Language
AUTHOR: H. P. Langtangen at Center for Biomedical Computing, Simula Research Laboratory and Dept. of Informatics, Univ. of Oslo
AUTHOR: Kaare Dump Email: dump@cyb.space.com at Segfault, Cyberspace Inc.
AUTHOR: A. Dummy Author
DATE: November 9, 2016
!ec
Note how one can specify a single institution, multiple institutions,
and no institution. In some formats (including `rst` and `sphinx`)
only the author names appear. Some formats have
"intelligence" in listing authors and institutions, e.g., the plain text
format:
!bc
Hans Petter Langtangen [1, 2]
Kaare Dump  (dump@cyb.space.com) [3]
A. Dummy Author

[1] Center for Biomedical Computing, Simula Research Laboratory
[2] Department of Informatics, University of Oslo
[3] Segfault, Cyberspace Inc.
!ec
Similar typesetting is done for LaTeX and HTML formats.

The current date can be specified as `today`.

idx{headlines} idx{section headings}

__Section Headings.__
Section headings are recognized by being surrounded by equal signs (=) or
underscores before and after the text of the headline. Different
section levels are recognized by the associated number of underscores
or equal signs (=):

   * 9 `=` characters for chapters
   * 7 for sections
   * 5 for subsections
   * 3 for subsubsections
   * 2 *underscrores* (only! - it looks best) for paragraphs
     (paragraph heading will be inlined)

Headings can be surrounded by as many blanks as desired.

Doconce also supports abstracts. This is typeset as a paragraph, but
*must* be followed by a section heading (everything up to the first
section heading is taken as part of the text of the abstract).


Here are some examples:
!bc
__Abstract.__ The following text just attempts to exemplify
various section headings.

========= Example on a Chapter Heading =========

Some text.


======= Example on a Section Heading =======

The running text goes here.


===== Example on a Subsection Heading =====

The running text goes here.

===Example on a Subsubsection Heading===

The running text goes here.

__A Paragraph.__ The running text goes here.
!ec

======= Special Lines =======

===== Figures =====

Figures are recognized by the special line syntax
!bc
FIGURE:[filename, height=xxx width=yyy scale=zzz] possible caption
!ec
The filename can be without extension, and Doconce will search for an
appropriate file with the right extension. If the extension is wrong,
say `.eps` when requesting an HTML format, Doconce tries to find another
file, and if not, the given file is converted to a proper format
(using ImageMagick's `convert` utility).

The height, width, and scale keywords (and others) can be included
if desired and may have effect for some formats. Note the comma
between the sespecifications and that there should be no space
around the = sign.

Note also that, like for `TITLE:` and `AUTHOR:` lines, all information
related to a figure line *must be written on the same line*. Introducing
newlines in a long caption will destroy the formatting (only the
part of the caption appearing on the same line as `FIGURE:` will be
included in the formatted caption).

FIGURE:[figs/streamtubes, width=400] Streamtube visualization of a fluid flow. label{fig:viz}

Combining several image files into one can be done by the
`convert` and `montage` programs from the ImageMagick suite:
!bc sys
montage file1.png file2.png ... file4.png -geometry +2+2  result.png
montage file1.png file2.png -tile x1 result.png
montage file1.png file2.png -tile 1x result.png

convert -background white file1.png file2.png +append tmp.png
!ec
Use `+append` for stacking left to right, `-append` for top to bottom.
The positioning of the figures can be controlled by `-gravity`.

===== Movies =====

Here is an example on the `MOVIE:` keyword for embedding movies. This
feature works well for the `latex`, `html`, `rst`, and `sphinx` formats.
Other formats try to generate some HTML file and link to that file
for showing the movie.
!bc
MOVIE: [filename, height=xxx width=yyy] possible caption
!ec

# latex/PDF format can make use of the movie15 package for displaying movies,
# or just plain \href{run: ...}{...}

MOVIE: [figs/mjolnir.mpeg, width=600 height=470]

#MOVIE: [figs/wavepacket.gif, width=600 height=470]

#MOVIE: [figs/wavepacket2.mpeg, width=600 height=470]

The LaTeX format results in a file that can either make use of
the movie15 package (requires the PDF to be shown in Acrobat Reader)
or just a plain address to the movie. The HTML, reST, and
Sphinx formats will play
the movie right away by embedding the file in a standard HTML code,
provided the output format is HTML.
For all other formats a URL to an HTML file, which can play the code,
is inserted in the output document.

When movies are embedded in the PDF file via LaTeX and
the `movie15` package wanted, one has to turn on the preprocessor
variable `MOVIE15`. There is an associated variable
`EXTERNAL_MOVIE_VIEWER` which can be defined to launch an external
viewer when displaying the PDF file (in Acrobat Reader):
!bc sys
Terminal> ptex2tex -DMOVIE15 -DEXTERNAL_MOVIE_VIEWER mydoc
!ec

The HTML, reST, and Sphinx formats can also treat filenames of the form
`myframes*.png`. In that case, an HTML file for showing the sequence of frames
is generated, and a link to this file is inserted in the output document.
That is, a simple "movie viewer" for the frames is made.

Many publish their scientific movies on YouTube, and Doconce recognizes
YouTube URLs as movies. When the output is an HTML file, the movie will
be embedded, otherwise a URL to the YouTube page is inserted.
You should equip the `MOVIE:` command with the right width and height
of embedded YouTube movies (the parameters appear when you request
the embedded HTML code for the movie on the YouTube page).



===== Copying Computer Code from Source Files =====

Another type of special lines starts with `@@@CODE` and enables copying
of computer code from a file directly into a verbatim environment, see
Section ref{sec:verbatim:blocks} below.


===== Inline Tagging =====
label{inline:tagging}
idx{inline tagging} idx{emphasized words} idx{boldface words} idx{verbatim text}
idx{inline comments}

Doconce supports tags for *emphasized phrases*, _boldface phrases_,
and `verbatim text` (also called type writer text, for inline code)
plus LaTeX/TeX inline mathematics, such as $\nu = \sin(x)$|$v = sin(x)$.

Emphasized text is typeset inside a pair of asterisk, and there should
be no spaces between an asterisk and the emphasized text, as in
!bc
*emphasized words*
!ec

Boldface font is recognized by an underscore instead of an asterisk:
!bc
_several words in boldface_ followed by *ephasized text*.
!ec
The line above gets typeset as
_several words in boldface_ followed by *ephasized text*.

Verbatim text, typically used for short inline code,
is typeset between back-ticks:
!bc
`call myroutine(a, b)` looks like a Fortran call
while `void myfunc(double *a, double *b)` must be C.
!ec
The typesetting result looks like this:
`call myroutine(a, b)` looks like a Fortran call
while `void myfunc(double *a, double *b)` must be C.

It is recommended to have inline verbatim text on the same line in
the Doconce file, because some formats (LaTeX and `ptex2tex`) will have
problems with inline verbatim text that is split over two lines.

Watch out for mixing back-ticks and asterisk (i.e., verbatim and
emphasized code): the Doconce interpreter is not very smart so inline
computer code can soon lead to problems in the final format. Go back to the
Doconce source and modify it so the format to which you want to go
becomes correct (sometimes a trial and error process - sticking to
very simple formatting usually avoids such problems).

Web addresses with links are typeset as
!bc
some URL like "Doconce": "http://code.google.com/p/doconce"
!ec
which appears as some URL like "Search Google": "http://google.com".
The space after colon is optional.
Links to files ending in `.txt`, `.html`, `.pdf`, `.py`, `.f`,
`.f77`, `.f90`, `.f95`, `.sh`, `.csh`, `.ksh`, `.zsh`,
`.c`, `.cpp`, `.cxx`, `.pl`, and `.java` follows the same
setup:
!bc
see the "Doconce Manual": "manual.do.txt".
!ec
which appears as see the "Doconce Manual": "manual.do.txt".
If you want to view such source code files *in the browser*,
rather than dowloading the file, we recommend to transform
the source code file to HTML format by running
`pygmentize`, e.g.,
!bc sys
Terminal> pygmentize -l bash -f html -O full,style=emacs \
          -o _static/make.sh.html subdir/make.sh
!ec
Then you can link to `_static/make.sh.html` instead of
`subdir/make.sh`. Or you can do both versions:
!bc
See the code URL:"src/myprog.py" ("view: "_static/myprog.py.html").
!ec

Links to files with other extensions must be realized
*with the filename as link text*, written as
the keyword URL, followed by a colon, and then the filename enclosed
in double quotes:
!bc
URL: "manual.html"
!ec
resulting in the link URL: "manual.html".

# This is now automatically carried out by the autogenerated
# script for sphinx:
#For such local links to
#work with the `sphinx` format, the `.rst` file needs a fix, carried
#out by
#!bc sys
#doconce sphinxfix_localURLs mydoc.rst
#!ec
#(The files, such as `manual.html`, are then copied to a subdirectory
#`_static`, which must be manually copied to the Sphinx directory's
#`_static` directory - links in the `.rst` files are automatically
#adjusted.)

To have the URL address itself as link text, put an "URL" or URL
before the address enclosed in double quotes:
!bc
Click on this link: URL:"http://code.google.com/p/doconce".
!ec
resulting in Click on this link: URL: "http://code.google.com/p/doconce".

Doconce also supports inline comments in the text:
!bc
[name: comment]
!ec
where `name` is the name of the author of the command, and `comment` is a
plain text text. [hpl: Note that there must be a space after the colon,
otherwise the comment is not recognized. Inline comments
can span
several lines,
if desired.]
The name and comment are visible in the output unless `doconce format`
is run with a command-line argument `--skip_inline_comments`
(see Chapter ref{doconce2formats} for an example). Inline comments
are helpful during development of a document since different authors
and readers can comment on formulations, missing points, etc.
All such comments can easily be removed from the `.do.txt` file
(see Chapter ref{doconce2formats}).

Inline mathematics is written as in LaTeX, i.e., inside dollar signs.
Most formats leave this syntax as it is (including to dollar signs),
hence nice math formatting is only obtained in LaTeX (Epytext has some
inline math support that is utilized).  However, mathematical
expressions in LaTeX syntax often contains special formatting
commands, which may appear annoying in plain text. Doconce therefore
supports an extended inline math syntax where the writer can provide
an alternative syntax suited for formats close to plain ASCII:
!bc
Here is an example on a linear system
${\bf A}{\bf x} = {\bf b}$|$Ax=b$,
where $\bf A$|$A$ is an $n\times n$|$nxn$ matrix, and
$\bf x$|$x$ and $\bf b$|$b$ are vectors of length $n$|$n$.
!ec
That is, we provide two alternative expressions, both enclosed in
dollar signs and separated by a pipe symbol, the expression to the
left is used in LaTeX, while the expression to the right is used for
all other formats.  The above text is typeset as "Here is an example
on a linear system ${\bf A}{\bf x} = {\bf b}$|$Ax=b$, where $\bf A$|$A$
is an $n\times n$|$nxn$ matrix, and $\bf x$|$x$ and $\bf b$|$b$
are vectors of length $n$|$n$."

===== Comments ======

Comments intended to be visible in the output document and read by
readers are known as *inline comments* in Doconce and described
in Section ref{inline:tagging}.

Here we address comments in the Doconce source file that are not
intended to be visible in the output document. Basic comment
lines start with the hash `#`:
!bc
#
# Here are some comment lines that do not affect any formatting.
# These lines are converted to comments in the output format.
#
!ec
Such comment lines may have some side effects in the `rst` and `sphinx`
formats because following lines are taken as part of the comment if
there is not a blank line after the comment.

The Mako preprocessor supports comments that are filtered out *before*
Doconce starts translating the document. Such comments are very valuable
as they will never interfere with the output format and they are only
present in the Doconce source. Mako has two types of comments:
lines starting with a double hash `##` and lines enclosed by
the `<%doc>` (beginning) and `<%doc/>` (closing) tags.

If you need a lot of comments in the Doconce file, consider using
Mako comments instead of the single hash, unless you want to
comments to be in the source code of the output document.


===== Cross-Referencing =====
idx{cross referencing} idx{labels} idx{references}

References and labels are supported. The syntax is simple:
!bc
label{section:verbatim}   # defines a label
For more information we refer to Section ref{section:verbatim}.
!ec
This syntax is close that that of labels and cross-references in
LaTeX. When the label is placed after a section or subsection heading,
the plain text, Epytext, and StructuredText formats will simply
replace the reference by the title of the (sub)section.  All labels
will become invisible, except those in math environments.  In the
`rst` and `sphinx` formats, the end effect is the same, but
the "label" and "ref" commands are first translated to the proper
reST commands by `doconce format`. In the HTML and (Google
Code) wiki formats, labels become anchors and references become links,
and with LaTeX "label" and "ref" are just equipped with backslashes so
these commands work as usual in LaTeX.

It is, in general, recommended to use labels and references for
(sub)sections, equations, and figures only.
By the way, here is an example on referencing Figure ref{fig:viz}
(the label appears in the figure caption in the source code of this document).
Additional references to Sections ref{mathtext} and ref{newcommands} are
nice to demonstrate, as well as a reference to equations,
say (ref{my:eq1})--(ref{my:eq2}). A comparison of the output and
the source of this document illustrates how labels and references
are handled by the format in question.

Hyperlinks to files or web addresses are handled as explained
in Section ref{inline:tagging}.

===== Index and Bibliography =====
idx{index} idx{citations} idx{bibliography}

An index can be created for the `latex`, `rst`, and `sphinx` formats
by the `idx` keyword, following a LaTeX-inspired syntax:
!bc
idx{some index entry}
idx{main entry!subentry}
idx{`verbatim_text` and more}
!ec
The exclamation mark divides a main entry and a subentry. Backquotes
surround verbatim text, which is correctly transformed in a LaTeX setting to
!bc
\index{verbatim\_text@\texttt{\rm\smaller verbatim\_text and more}}
!ec
Everything related to the index simply becomes invisible in plain
text, Epytext, StructuredText, HTML, and wiki formats.  Note: `idx`
commands should be inserted outside paragraphs, not in between the
text as this may cause some strange behaviour of reST and
Sphinx formatting.  As a recommended rule, index items are naturally
placed right after section headings, before the text begins, while
index items related to a paragraph should be placed above the
paragraph one a separate line (and not in between the text or between
the paragraph heading and the text body, although this works fine if
LaTeX is the output format).

Literature citations also follow a LaTeX-inspired style:
!bc
as found in cite{Larsen_1986,Nielsen_Kjeldstrup_1999}.
!ec
Citation labels can be separated by comma. In LaTeX, this is directly
translated to the corresponding `cite` command; in reST
and Sphinx the labels can be clicked, while in all the other text
formats the labels are consecutively numbered so the above citation
will typically look like
!bc
as found in [3][14]
!ec
if `Larsen_1986` has already appeared in the 3rd citation in the document
and `Nielsen_Kjeldstrup_1999` is a new (the 14th) citation. The citation labels
can be any sequence of characters, except for curly braces and comma.

The bibliography itself is specified by the special keyword `BIBFILE:`,
which is optionally followed by a BibTeX file, having extension `.bib`,
a corresponding reST bibliography, having extension `.rst`,
or simply a Python dictionary written in a file with extension `.py`.
The dictionary in the latter file should have the citation labels as
keys, with corresponding values as the full reference text for an item
in the bibliography. Doconce markup can be used in this text, e.g.,
!bc
{
'Nielsen_Kjeldstrup_1999': """
K. Nielsen and A. Kjeldstrup. *Some Comments on Markup Languages*.
URL:"http://some.where.net/nielsen/comments", 1999.
""",
'Larsen_1986':
"""
O. B. Larsen. On Markup and Generality.
*Personal Press*. 1986.
"""
}
!ec
In the LaTeX format, the `.bib` file will be used in the standard way,
in the `rst` and `sphinx` formats, the `.rst` file will be
copied into the document at the place where the `BIBFILE:` keyword
appears, while all other formats will make use of the Python dictionary
typeset as an ordered Doconce list, replacing the `BIBFILE:` line
in the document.

At present, only one file with bibliographic references can be used.

# see ketch/tex2rst for nice bibtex to rst converter which could
# be used here

Conversion of BibTeX databases to reST format can be
done by the "bibliograph.parsing":"http://pypi.python.org/pypi/bibliograph.parsing/" tool.

Finally, we here test the citation command and bibliography by
citing a book cite{Python:Primer:09}, a paper cite{Osnes:98},
and both of them simultaneously cite{Python:Primer:09,Osnes:98}.

[somereader: comments, citations, and references in the latex style
is a special feature of doconce :-) ]


===== Tables =====

A table like

  |--------------------------------|
  |time  | velocity | acceleration |
  |--r--------r-----------r--------|
  | 0.0  | 1.4186   | -5.01        |
  | 2.0  | 1.376512 | 11.919       |
  | 4.0  | 1.1E+1   | 14.717624    |
  |--------------------------------|

is built up of pipe symbols and dashes:
!bc
  |--------------------------------|
  |time  | velocity | acceleration |
  |--------------------------------|
  | 0.0  | 1.4186   | -5.01        |
  | 2.0  | 1.376512 | 11.919       |
  | 4.0  | 1.1E+1   | 14.717624    |
  |--------------------------------|
!ec
The pipes and column values do not need to be aligned (but why write
the Doconce source in an ugly way?). In the line below the heading,
one can insert the characters `c`, `r`, or `l` to specify the
alignment of the columns (centered, right, or left, respectively).
Similar character can be inserted in the line above the header to
algn the headings. Pipes `|` can also be inserted to indicate
vertical rules in LaTeX tables (they are ignored for other formats).
Note that not all formats offer alignment of heading or entries
in tables (`rst` and `sphinx` are examples). Also note that
Doconce tables are very simple: neither entries nor
headings can span several columns or rows. When that functionality
is needed, one can make use of the preprocessor and if-tests on
the format and insert format-specific code for tables.



===== Exercises, Problems, or Projects =====

Doconce has special support for three types of "exercises", named
*exercise*, *problem*, or *project*.
These are all typeset as special kind of
sections. Such sections start with a subsection or subsubsection
headline, indicated by 3 or 5 `=` characters, and last up to the
next headline or the end of the file. The headline itself must
consists of the word "Exercise", "Problem", or "Project", followed
by a colon and a title of the exercise, problem, or project.
The next line(s) may contain a label and specification of the
name of result file (if the answer to the exercise is to be handed
it) and a solution file:
!bc
===== Project: Determine the Distance to the Moon =====
label{proj:moondist} file=earth2moon.pdf
solution=eart2moon_sol.do.txt

Here goes the running text of the project....

__Hint 1.__ Do not plan a travel to the moon.

__Hint 2.__ Wikipedia is always helpful.
!ec
Doconce will recognize the exercise, problem, or project *title*,
the *label*, the *result file*, the *solution file* (if any of
these three entities is present), the *text*, and a sequence of
*hints*. Tailored formatting of exercises in special output formats
can make use of this. For example, one can image web formats where
the hints are displayed one by one when needed and where the result
file can be uploaded. One can also think of mechanisms for downloading
the solution file if the result file meets certain criteria.
Doconce does not yet generate such functionality in any output format,
but this is an intended future feature to be impelemented.

Because exercises, problems, and projects are typeset as ordinary
sections (this is the most general approach that will work for many
formats), one must refer to an exercise, problem, or project
by its label, which normally will translate to the section number
(in LaTeX, for instance) or a link to the title of the section.
The *title* is typeset without any leading "Exercise:", "Problem:",
or "Project:" word, so that references like "see Problem ..."
works well in all formats ("..." will be a number in LaTeX and
the problem title in most other formats).

It is recommended to collect all exercises as subsetions (or subsubsections)
under a section (or subsection) named "Exercises", "Problems", or
"Projects".


===== Blocks of Verbatim Computer Code =====
label{sec:verbatim:blocks}

Blocks of computer code, to be typeset verbatim, must appear inside a
"begin code" `!bc` keyword and an "end code" `!ec` keyword. Both
keywords must be on a single line and *start at the beginning of the
line*.  Before such a code block there must be a plain sentence
(at least if successful transformation to reST and
ASCII-type formats is desired). For example, a code block cannot come
directly after a section/paragraph heading or a table.

There may be an argument after the `!bc` tag to specify a
certain environment (for `ptex2tex` or Sphinx) for typesetting
the verbatim code. For instance, `!bc dat` corresponds to
the data file environment and `!bc cod` is typically
used for a code snippet. There are some predefined environments
explained below. If there is
no argument specifying the environment, one assumes some plain
verbatim typesetting (for `ptex2tex` this means the `ccq` environment,
which is defined in the config file `.ptex2tex.cfg`,
while for Sphinx it defaults to the `python` environment).

Since the config file for `ptex2tex` can define what some environment
maps onto with respect to typesetting, a similar possibility is
supported for Sphinx as well.  The argument after `!bc` is in case of
Sphinx output mapped onto a valid Pygments language for typesetting of
the verbatim block by Pygments. This mapping takes place in an
optional comment to be inserted in the Doconce source file, e.g.,
!bc
# sphinx code-blocks: pycod=python cod=fortran cppcod=c++ sys=console
!ec
Here, three arguments are defined: `pycod` for Python code,
`cod` also for Python code, `cppcod` for C++ code, and `sys`
for terminal sessions. The same arguments would be defined
in `.ptex2tex.cfg` for how to typeset the blocks in LaTeX using
various verbatim styles (Pygments can also be used in a LaTeX
context).

By default, `pro` is used for complete programs in Python, `cod` is
for a code snippet in Python, while `xcod` and `xpro` implies computer
language specific typesetting where `x` can be `f` for Fortran, `c`
for C, `cpp` for C++, `sh` for Unix shells, `pl` for Perl, `m` for
Matlab, `cy` for Cython, and `py` for Python.  The argument `sys`
means by default `console` for Sphinx and `CodeTerminal` (ptex2tex
environent) for LaTeX. Other specifications are `dat` for a data file
or print out, and `ipy` for interactive Python sessions (the latter
does not introduce any environment  in `sphinx` output, as interactive
sessions are automatically recognized and handled).  All these
definitions of the arguments after `!bc` can be redefined in the
`.ptex2tex.cfg` configuration file for ptex2tex/LaTeX and in the
`sphinx code-blocks` comments for Sphinx. Support for other languages
is easily added.

# (Any sphinx code-block comment, whether inside verbatim code
# blocks or outside, yields a mapping between bc arguments
# and computer languages. In case of muliple definitions, the
# first one is used.)

The enclosing `!ec` tag of verbatim computer code blocks must
be followed by a newline.  A common error in list environments is to
forget to indent the plain text surrounding the code blocks. In
general, we recommend to use paragraph headings instead of list items
in combination with code blocks (it usually looks better, and some
common errors are naturally avoided).

Here is a verbatim code block with Python code (`pycod` style):
!bc pycod
# regular expressions for inline tags:
inline_tag_begin = r'(?P<begin>(^|\s+))'
inline_tag_end = r'(?P<end>[.,?!;:)\s])'
INLINE_TAGS = {
    'emphasize':
    r'%s\*(?P<subst>[^ `][^*`]*)\*%s' % \
    (inline_tag_begin, inline_tag_end),
    'verbatim':
    r'%s`(?P<subst>[^ ][^`]*)`%s' % \
    (inline_tag_begin, inline_tag_end),
    'bold':
    r'%s_(?P<subst>[^ `][^_`]*)_%s' % \
    (inline_tag_begin, inline_tag_end),
}
!ec
And here is a C++ code snippet (`cppcod` style):
!bc cppcod
void myfunc(double* x, const double& myarr) {
    for (int i = 1; i < myarr.size(); i++) {
        myarr[i] = myarr[i] - x[i]*myarr[i-1]
    }
}
!ec

# When showing copy from file in !bc envir, intent a character - otherwise
# ptex2tex is confused and starts copying...
Computer code can be copied directly from a file, if desired. The syntax
is then
!bc
 @@@CODE myfile.f
 @@@CODE myfile.f fromto: subroutine\s+test@^C\s{5}END1
!ec
The first line implies that all lines in the file `myfile.f` are
copied into a verbatim block, typset in a `!bc Xpro` environment, where
`X` is the extension of the filename, here `f` (i.e., the environment
becomes `!bc fpro` and will typically lead to some Fortran-style
formatting in Linux and Sphinx).  The
second line has a `fromto:` directive, which implies copying code
between two lines in the code, typset within a !`bc Xcod`
environment (again, `X` is the filename extension, implying the
type of file). Note that the `pro` and `cod` arguments are only used for LaTeX
and Sphinx output, all other formats will have the code typeset within
a plain `!bc` environment.) Two regular expressions, separated by the
`@` sign, define the "from" and "to" lines.  The "from" line is
included in the verbatim block, while the "to" line is not. In the
example above, we copy code from the line matching `subroutine test`
(with as many blanks as desired between the two words) and the line
matching `C END1` (C followed by 5 blanks and then the text END1). The
final line with the "to" text is not included in the verbatim block.

Let us copy a whole file (the first line above):

@@@CODE __testcode.f

Let us then copy just a piece in the middle as indicated by the `fromto:`
directive above:

@@@CODE __testcode.f fromto:subroutine\s+test@^C\s{5}END1

Note that the "to" line is not copied into the Doconce file, but the
"from" line is. Sometimes it is convenient to also neglect the
"from" line, a feature that is allowed by replacing `fromto:` by
`from-to` ("from with minus"). This allows for copying very similar
code segments throughout a file, while still distinguishing between them.
Copying the second set of parameters from the text
!bc cod
# --- Start Example 1 ---
c = -1
A = 2
p0 = 4
simulate_and_plot(c, A, p0)
# --- End Example 1 ---

# --- Start Example 2 ---
c = -1
A = 1
p0 = 0
simulate_and_plot(c, A, p0)
# --- End Example 2 ---
!ec
is easy with
!bc
from-to: Start Example 2@End Example 2
!ec
With only `fromto:` this would be impossible.

(Remark for those familiar with `ptex2tex`: The from-to
syntax is slightly different from that used in `ptex2tex`. When
transforming Doconce to LaTeX, one first transforms the document to a
`.p.tex` file to be treated by `ptex2tex`. However, the `@@@CODE` line
is interpreted by Doconce and replaced by the mentioned
pro or cod environment which are defined in the `ptex2tex` configuration
file.)


===== LaTeX Blocks of Mathematical Text =====
label{mathtext}

Blocks of mathematical text are like computer code blocks, but
the opening tag is `!bt` (begin TeX) and the closing tag is
`!et`. It is important that `!bt` and `!et` appear on the beginning of the
line and followed by a newline.
!bc
!bt
\begin{align}
{\partial u\over\partial t} &= \nabla^2 u + f, label{myeq1}\\
{\partial v\over\partial t} &= \nabla\cdot(q(u)\nabla v) + g
\end{align}
!et
!ec
Here is the result of the above `!bt` - `!et` block:
!bt
\begin{align}
{\partial u\over\partial t} &= \nabla^2 u + f, label{myeq1}\\
{\partial v\over\partial t} &= \nabla\cdot(q(u)\nabla v) + g
\end{align}
!et

This text looks ugly in all Doconce supported formats, except from
LaTeX and Sphinx.  If HTML is desired, and Sphinx is not an option,
one can filter the Doconce text first to LaTeX and then use the tex4ht
tool to convert the dvi file to HTML, or one could just link a PDF
file (made from LaTeX) directly from HTML.

If the document targets both LaTeX, Sphinx, and other formats like
plain text and wiki, one can use the preprocessor to typeset the
mathematics in two versions. After `#if FORMAT in ("latex", "pdflatex",
"sphinx", "mwiki")` one places LaTeX mathematics, and after `#else` one can
write inline mathematics in a way that looks nice in plain text and
wiki formats without support for mathematical typesetting.


=== Mathematics for PowerPoint/OpenOffice ===

If you have LaTeX mathematics written in Doconce, it is fairly easy
to generate PNG images of all mathematical formulas and equations for
use with PowerPoint or OpenOffice documents.

  o Make a Sphinx version of the Doconce file.
  o Go to the Sphinx directory and load the `conf.py` file into
    a browser.
  o Search for "math" and comment out the
    `'sphinx.ext.mathjax'` (enabled by default) and
    `'matplotlib.sphinxext.mathmpl'` (disabled by default)
    lines, and uncomment the `'sphinx.ext.pngmath'` package.
    This is the package that generates small PNG pictures
    of the mathematics.
  o Uncomment the line with `pngmath_dvipng_args =` and
    set the PNG resolution to `-D 200` when the purpose is to
    generate mathematics pictures for slides.
  o Run `make html`.
  o Look at the HTML source file in the `_build/html`
    directory: all mathematics are in `img` tags with `src=`
    pointing to a PNG file and `alt=` pointing to the LaTeX
    source for the formula in question. This makes it very
    easy to find the PNG file that corresponding to a particular
    mathematical expression.


===== Macros (Newcommands) =====
label{newcommands}

Doconce supports a type of macros via a LaTeX-style *newcommand*
construction.  The newcommands defined in a file with name
`newcommand_replace.tex` are expanded when Doconce is filtered to
other formats, except for LaTeX (since LaTeX performs the expansion
itself).  Newcommands in files with names `newcommands.tex` and
`newcommands_keep.tex` are kept unaltered when Doconce text is
filtered to other formats, except for the Sphinx format. Since Sphinx
understands LaTeX math, but not newcommands if the Sphinx output is
HTML, it makes most sense to expand all newcommands.  Normally, a user
will put all newcommands that appear in math blocks surrounded by
`!bt` and `!et` in `newcommands_keep.tex` to keep them unchanged, at
least if they contribute to make the raw LaTeX math text easier to
read in the formats that cannot render LaTeX.  Newcommands used
elsewhere throughout the text will usually be placed in
`newcommands_replace.tex` and expanded by Doconce.  The definitions of
newcommands in the `newcommands*.tex` files *must* appear on a single
line (multi-line newcommands are too hard to parse with regular
expressions).

__Example.__ Suppose we have the following commands in
`newcommand_replace.tex`:

@@@CODE newcommands_replace.tex

and these in `newcommands_keep.tex`:

@@@CODE newcommands_keep.tex

The LaTeX block
!bc
\beqa
\x\cdot\normalvec &=& 0, label{my:eq1}\\
\Ddt{\uvec} &=& \Q \ep   label{my:eq2}
\eeqa
!ec
will then be rendered to
!bt
\beqa
\x\cdot\normalvec &=& 0, label{my:eq1}\\
\Ddt{\uvec} &=& \Q \ep   label{my:eq2}
\eeqa
!et
in the current format.

===== Preprocessing Steps =====

Doconce allows preprocessor commands for, e.g., including files,
leaving out text, or inserting special text depending on the format.
Two preprocessors are supported: preprocess
(URL:"http://code.google.com/p/preprocess") and mako
(URL:"http://www.makotemplates.org/"). The former allows include and if-else
statements much like the well-known preprocessor in C and C++ (but it
does not allow sophisticated macro substitutions). The latter
preprocessor is a very powerful template system.  With Mako you can
automatically generate various type of text and steer the generation
through Python code embedded in the Doconce document. An arbitrary set
of `name=value` command-line arguments (at the end of the command line)
automatically define Mako variables that are substituted in the document.

Doconce will detect if preprocess or Mako commands are used and run
the relevant preprocessor prior to translating the Doconce source to a
specific format.

The preprocess and mako programs always have the variable `FORMAT`
defined as the desired output format of Doconce (`html`, `latex`,
`plain`, `rst`, `sphinx`, `epydoc`, `st`).  It is then easy to test on
the value of `FORMAT` and take different actions for different
formats. For example, one may create special LaTeX output for figures,
say with multiple plots within a figure, while other formats may apply
a separate figure for each plot. Below is an example (see the Doconce
source code of this document to understand how preprocess is used to
create the example).

# If PNGFIGS is defined, PNG files are used, otherwise Encapsulated
# PostScript files are used.

# #if FORMAT == "latex"
# Use latex with subfigures (a) and (b)
\begin{figure}
label{fig:wavepackets}
  \begin{center}
#  #ifdef PNGFIGS
\subfigure[]{\includegraphics[width=0.49\linewidth]{figs/wavepacket_0001.png}}
#  #else
\subfigure[]{\includegraphics[width=0.49\linewidth]{figs/wavepacket_0001.eps}}
#  #endif

#  #ifdef PNGFIGS
\subfigure[]{\includegraphics[width=0.49\linewidth]{figs/wavepacket_0010.png}}
#  #else
\subfigure[]{\includegraphics[width=0.49\linewidth]{figs/wavepacket_0010.eps}}
#  #endif
  \end{center}
  \caption{
  Wavepackets at time (a) 0.1 s and (b) 0.2 s.
  }
\end{figure}

# #else

# Use default Doconce figure handling for all other formats

FIGURE:[figs/wavepacket_0001.png, width=400] Wavepacket at time 0.1 s.

FIGURE:[figs/wavepacket_0010.png, width=400] Wavepacket at time 0.2 s.

# #endif

Other user-defined variables for the preprocessor can be set at
the command line as explained in Section ref{doconce2formats}.

More advanced use of mako can include Python code that may automate
the writing of parts of the document.



===== Splitting Documents into Smaller Pieces =====

Long documents are conveniently split into smaller Doconce files.
However, there must be a master document including all the pieces,
otherwise references to sections and the index will not work properly.
The master document is preferably a file just containing a set of
preprocessor include statements of the form `#include "file.do.txt"`.
The preprocessor will put together all the pieces so that Doconce
sees a long file with the complete text.

For reST and Sphinx documents it is a point to have
separate `.rst` files and an index file listing the various `.rst`
that build up the document. To generate the various `.rst` files one
should not run Doconce on the individual `.do.txt` files, because then
references and index entries are not treated correctly. Instead,
run Doconce on the master file and invoke the script `doconce split_rst`
to split the long, complete `.rst` into pieces. This process requires
that each `#include "file.do.txt` line in the master file is preceded by a
"marker line" having the syntax `#>>>>>> part: file >>>>>>`, where
`file` is the filename without extension. The number of greater than
signs is not important, but it has to be a comment line and it has
to contain the keyword `part:`.

Here is an example. Say the name of the master file is `master.do.txt`.
The following Bash script does the job:
We run
!bc cod
doconce format sphinx master
# Split master.rst into parts
# as defined by #>>>>> part: name >>>>> lines
files=`doconce split_rst master.rst`

dir=sphinxm-rootdir

if [ ! -d $dir ]; then
  doconce sphinx_dir dirname=$dir author='me and you' \
          version=1.0 theme=default $files
  sh automake-sphinx.sh
else
  for file in $files; do
    cp $file.rst $dir
  done
  cd $dir
  make html
  cd ..
fi
!ec
The autogenerated `automake-sphinx.sh` file (by `doconce sphinx_dir`)
is compatible with a master `.rst` file split into pieces as long as
the complete set of pieces in correct order is given to `doconce sphinx_dir`.
This set is the output of `doconce split_rst`, which we catch in a
variable `files` above.


===== Missing Features =====

Doconce does not aim to support sophisticated typesetting, simply because
sophisticated typesetting usually depend quite strongly on the particular
output format chosen. When a particular feature needed is not supported
by Doconce, it is recommended to hardcode that feature for a particular
format and use the if-else construction of the preprocessor. For example,
if a sophisticated table is desired in LaTeX output, test on the
`FORMAT` variable (`#if FORMAT == "latex"`) in the Doconce code and
insert the table in LaTeX format. The `else` or `elif` branches may
have the table in other formats or a simplified version in the Doconce
table format. Similarly, if certain adjustments are needed, like
pagebreaks in LaTeX, hardcode that in the Doconce format (and recall
that this is really LaTeX dependent - pagebreaks are not
relevant HTML formats).

Instead of inserting special code in the Doconce document, one can
alternatively script editing of the output from Doconce. That is,
we develop a Python or Bash script that runs the translation of
a Doconce document to a ready docoment in another format. Inside this
script, we may edit and fine-tune the output from Doconce.

As an example, say you want a table of contents in the LaTeX output
(Doconce does not support table of contents). By inserting a
recognizable comment in the Doconce source, say
!bc
# table of contents
!ec
we can use this comment to edit the LaTeX file. First, we run
Doconce `doconce format latex mydoc` to produce `mydoc.p.tex`. Then
we use the `doconce replace` and `doconce subst` commands to
replace the comment by the comment plus the table of contents command,
or just the latter:
!bc
Terminal> doconce replace '% table of contents'
          '\tableofcontents' mydoc.p.tex
!ec
The `doconce replace from_text to_text filename` command performs a
character-by-character replacement (using the `replace` method in
string objects in Python). If we want to preserve the comment and add
a new line with `\tableofcontents`, we should use `doconce subst`,
which applies regular expressions for substitutions and thereby
understands the newline character:
!bc
Terminal> doconce subst '% table of contents' \
          '% table of contents\n\\tableofcontents' mydoc.p.tex
!ec
Note the double backshlash in front of the `t` character: without it we
would get a tab and no backslash.
The `doconce subst` is a powerful way to automatically edit the output
from Doconce and fine-tune a LaTeX document. Use of comment lines to
identify portions of the file to be edited is a smart idea.
Alternatively, the relevant LaTeX constructions can be inserted directly
in the Doconce file using if-else preprocessor directives.

===== Header and Footer =====

Some formats use a header and footer in the document. LaTeX and
HTML are two examples of such formats. When the document is to be
included in another document (which is often the case with
Doconce-based documents), the header and footer are not wanted, while
these are needed (at least in a LaTeX context) if the document is
stand-alone. We have introduce the convention that if `TITLE:` or
`#TITLE:` is found at the beginning of the line (i.e., the document
has, or has an intention have, a title), the header and footer
are included, otherwise not.


===== Emacs Doconce Formatter =====

The file `misc/.doconce-mode.el` in the Doconce source distribution
gives a "Doconce Editing Mode" in Emacs. The file is a rough edit of
the reST Editing Mode for Emacs. Some Doconce features are recognized,
but far from all, and sometimes portions of Doconce text just appear
as ordinary text.

Here is how to get the Doconce Editing Mode in Emacs.

__Step 1.__ Download the Doconce tarball from `code.google.com/p/doconce`,
pack it out and go to the root directory.

__Step 2.__ Copy the `doconce-mode.el` file to the home directory:
!bc
cp misc/.doconce-mode.el $HOME
!ec

__Step 3.__ Add these lines to `$HOME/.emacs`:
!bc
(load-file "~/hg/.doconce-mode.el")
(setq auto-mode-alist(cons '("\\.do\\.txt$" . doconce-mode) auto-mode-alist))
!ec
Emacs will now recognize files with extension `.do.txt` and enter
the Doconce Editing Mode.


======= Troubleshooting =======

===== Disclaimer =====

Doconce has some support for syntax checking.  If you encounter Python
errors while running `doconce format`, the reason for the error is
most likely a syntax problem in your Doconce source file. You have to
track down this syntax problem yourself.

However, the problem may well be a bug in Doconce. The Doconce
software is incomplete, and many special cases of syntax are not yet
discovered to give problems. Such special cases are also seldom easy to
fix, so one important way of "debugging" Doconce is simply to change
the formatting so that Doconce treats it properly. Doconce is very much
based on regular expressions, which are known to be non-trivial to
debug years after they are created. The main developer of Doconce has
hardly any time to work on debugging the code, but the software works
well for his diverse applications of it.


===== General Problems ======

=== Something goes wrong in the preprocessing step ===

Doconce automatically removes the file `__tmp.do.txt`, which is the
resulting of the preprocessing stge and the file to examine if
something goes wrong in this stage (i.e., when `mako` and/or
`preprocess` is run). Add the `--debug` flag at the end of the
`doconce` command to (both make a debug file and) avoid that
`__tmp.do.txt` is deleted.

=== Figure captions are incomplete ===

If only the first part of a figure caption in the Doconce file is seen
in the target output format, the reason is usually that the caption
occupies multiple lines in the Doconce file. The figure caption must
be written as *one line*, at the same line as the FIGURE keyword.

=== Preprocessor directives do not work ===

Make sure the preprocessor instructions, in Preprocess or Mako, have
correct syntax. Also make sure that you do not mix Preprocess and Mako
instructions. Doconce will then only run Preprocess.

=== Problems with boldface and emphasize ===

Two boldface or emphasize expressions after each other are not rendered
correctly. Merge them into one common expression.

=== Links to local directories do not work ===

Links of the type
!bc
see the "examples directory": "src/examples"
!ec
do not work well. You need to link to a specific HTML file:
!bc
see the "examples directory": "src/examples/index.html"
!ec

=== Links are not typeset correctly ===

Not all formats will allow formatting of the links. Verbatim words
in links are allowed if the whole link is typeset in verbatim:
!bc
see the directory "`examples`": "src/examples/index.html".
!ec
However, the following will not be typeset correctly:
!bc
see the "`examples` directory": "src/examples/index.html"
!ec
The back-ticks must be removed, or the text can be reformulated as
in the line above it.


=== Inline verbatim code is not detected ===

Make sure there is a space before the first back-tick.

=== Strange non-English characters ===

Check the encoding of the `.do.txt` file with the Unix `file` command
or with
!bc
Unix> doconce guess_encoding myfile.do.txt
!ec
If the encoding is utf-8, convert to latin-1 using either of
the Unix commands
!bc
Unix> doconce change_encoding utf-8 LATIN1 myfile.do.txt

Unix> iconv -f utf-8 -t LATIN1 myfile.do.txt --output newfile
!ec

=== Wrong Norwegian charcters ===

When Doconce documents have characters not in the standard ASCII set,
the format of the file must be LATIN1 and not UTF-8. See
the section "Strange non-English characters" above for how to
run `doconce change_encoding` to change the encoding of the Doconce file.

=== Inline verbatim text is not formatted correctly ===

Make sure there is whitespace surrounding the text in back-ticks.

=== Too short underlining of reST headlines ===

This may happen if there is a paragraph heading without
proceeding text before some section heading.


=== Found !bt but no tex blocks extracted (BUG) ===

This message points to a bug, but has been resolved by removing blank lines
between the text and the first `!bt` (inserting the blanks again did not
trigger the error message again...).

===== Problems with code or Tex Blocks =====

=== Code or math block errors in reST ===

First note that a code or math block must come after some plain
sentence (at least for successful output in reST), not directly
after a section/paragraph heading, table, comment, figure, or
movie, because the code or math block is indented and then become
parts of such constructions. Either the block becomes invisible or
error messages are issued.

Sometimes reST reports an "Unexpected indentation" at the beginning of
a code block. If you see a `!bc`, which should have been removed when
running `doconce format sphinx`, it is usually an error in the Doconce
source, or a problem with the rst/sphinx translator.  Check if the
line before the code block ends in one colon (not two!), a question
mark, an exclamation mark, a comma, a period, or just a newline/space
after text. If not, make sure that the ending is among the
mentioned. Then `!bc` will most likely be replaced and a double colon
at the preceding line will appear (which is the right way in reST to
indicate a verbatim block of text).

=== Strange errors around code or TeX blocks in reST ===

If `idx` commands for defining indices are placed inside paragraphs,
and especially right before a code block, the reST translator
(rst and sphinx formats) may get confused and produce strange
code blocks that cause errors when the reST text is transformed to
other formats. The remedy is to define items for the index outside
paragraphs.

=== Something is wrong with a verbatim code block ===

Check first that there is a "normal" sentence right before
the block (this is important for reST and similar
"ASCII-close" formats).

=== Code/TeX block is not shown in reST format ===

A comment right before a code or tex block will treat the whole
block also as a comment. It is important that there is normal
running text right before `!bt` and `!bc` environments.

=== Verbatim code blocks inside lists look ugly ===

Read the Section ref{sec:verbatim:blocks} above.  Start the
`!bc` and `!ec` tags in column 1 of the file, and be careful with
indenting the surrounding plain text of the list item correctly. If
you cannot resolve the problem this way, get rid of the list and use
paragraph headings instead. In fact, that is what is recommended:
avoid verbatim code blocks inside lists (it makes life easier).

=== LaTeX code blocks inside lists look ugly ===

Same solution as for computer code blocks as described in the
previous paragraph. Make sure the `!bt` and `!et` tags are in column 1
and that the rest of the non-LaTeX surrounding text is correctly indented.
Using paragraphs instead of list items is a good idea also here.


===== Problems with reST/Sphinx Output ======

=== Lists do not appear in .rst files ===

Check if you have a comment right above the list. That comment
will include the list if the list is indentend. Remove the comment.

=== Error message "Undefined substitution..." from reST ===

This may happen if there is much inline math in the text. reST cannot
understand inline LaTeX commands and interprets them as illegal code.
Just ignore these error messages.

=== Warning about duplicate link names ===

Link names should be unique, but if (e.g.) "file" is used as link text
several places in a reST file, the links still work. The warning can
therefore be ignorned.

=== Inconsistent headings in reST ===

The `rst2*.py` and Sphinx converters abort if the headers of sections
are not consistent, i.e., a subsection must come under a section,
and a subsubsection must come under a subsection (you cannot have
a subsubsection directly under a section). Search for `===`,
count the number of equality signs (or underscores if you use that)
and make sure they decrease by two every time a lower level is encountered.

=== No code environment appears before "bc ipy" blocks ===

The `!bc ipy` directive behaves this way for `sphinx` output because
interactive sessions are automatically handled. If this is not
appropriate, shift to `!bc cod` or another specification of the
verbatim environment.

===== Problems with LaTeX Output ======

=== Error when running latex: You must 'pygmentize' installed ===

This points to the used of the Minted style for typesetting verbatim
code. You need to run
!bc sys
Terminal> latex -shell-escape file
!ec
if you have run `ptex2tex` with the `-DMINTED` option and turned on
the use of the Minted style (and therefore `pygmentize` program).

=== The LaTeX file does not compile ===

If the problem is undefined control sequence involving
!bc
\code{...}
!ec
the cause is usually a verbatim inline text (in back-ticks in the
Doconce file) spans more than one line. Make sure, in the Doconce source,
that all inline verbatim text appears on the same line.

=== Inline verbatim gives error  ===

Check if the inline verbatim contains typical LaTeX commands, e.g.,
!bc
some text with `\usepackage{mypack}` is difficult because
ptex2tex will replace this by \code{\usepackage{mypack}} and
then replace this by
{\fontsize{10pt}{10pt}\verb!\usepackage{mypack!}}
which is wrong because ptex2tex applies regex that don't
capture the second }
!ec
The remedy is to place verbatim LaTeX commands in verbatim
blocks - that is safe.

# Could have doconce configure file where inline verbatim is
# configured to be \fontsize... directly, not via ptex2tex \code{}.


=== Errors in figure captions ===

Such errors typically arise from unbalanced curly braces, or dollar signs
around math, and similar LaTeX syntax errors.

(Note that verbatim font is likely to cause trouble inside figure captions,
but Doconce will automatically replace verbatim text in back-ticks by
a proper `texttt` command (since verbatim font constructions does not work
inside figure captions) and precede underscores by backslash.)

=== Chapters are ignored ===

The default LaTeX style is "article". If you chapters in the Doconce file,
you need to run `ptex2tex` with the option `-DBOOK` to set the LaTeX
documentstyle to "book".

=== I want to tune the top of the LaTeX file ===

The top of the LaTeX file, as generated by Doconce, is very simple.
If this LaTeX code is not sufficient for your needs, there are
two ways out of it:

 o Make a little Bash script that performs a series of
   `doconce subst` (regular expressions) or `doconce replace` (regular text)
   substitutions to change the text automatically (you probably have to
   repeat these edits so automating them is a good idea).
 o Place the title, author(s), and date of the Doconce file in a separate
   file and use the preprocessor to include the rest. The rest is then
   one or more Doconce files without title, author(s), and date. This
   means that the `doconce format latex` command does not generate
   the LaTeX intro (preamble) and outro, just the core text, for these
   files.
   Make a new file by hand with the appropriate LaTeX intro and outro
   text and include the various text pieces in this file.
   To make the LaTeX document, you compile all Doconce files
   into LaTeX code, except the "top" Doconce file that includes the
   others. That file is not used for LaTeX output, but
   replaced by the hand-written LaTeX "top" file.

===== Problems with gwiki Output ======

=== Strange nested lists in gwiki ===

Doconce cannot handle nested lists correctly in the gwiki format.
Use nonnested lists or edit the `.gwiki` file directly.

=== Lists in gwiki look ugly in the gwiki source ===

Because the Google Code wiki format requires all text of a list item to
be on one line, Doconce simply concatenates lines in that format,
and because of the indentation in the original Doconce text, the gwiki
output looks somewhat ugly. The good thing is that this gwiki source
is seldom to be looked at - it is the Doconce source that one edits
further.

===== Debugging =====

Given a problem, extract a small portion of text surrounding the
problematic area and debug that small piece of text. Doconce does a
series of transformations of the text. The effect of each of these
transformation steps are dumped to a logfile, named
`_doconce_debugging.log`, if the to `doconce format` after the filename
is `debug`. The logfile is inteded for the developers of Doconce, but
may still give some idea of what is wrong.  The section "Basic Parsing
Ideas" explains how the Doconce text is transformed into a specific
format, and you need to know these steps to make use of the logfile.



======= Basic Parsing Ideas =======

# avoid list here since we have code in between (never a good idea)

The (parts of) files with computer code to be directly included in
the document are first copied into verbatim blocks.

All verbatim and TeX blocks are removed and stored elsewhere
to ensure that no formatting rules are not applied to these blocks.

The text is examined line by line for typesetting of lists, as well as
handling of blank lines and comment lines.
List parsing needs some awareness of the context.
Each line is interpreted by a regular expression

!bc
(?P<indent> *(?P<listtype>[*o-] )? *)(?P<keyword>[^:]+?:)?(?P<text>.*)\s?
!ec

That is, a possible indent (which we measure), an optional list
item identifier, optional space, optional words ended by colon,
and optional text. All lines are of this form. However, some
ordinary (non-list) lines may contain a colon, and then the keyword
and text group must be added to get the line contents. Otherwise,
the text group will be the line.

When lists are typeset, the text is examined for sections, paragraphs,
title, author, date, plus all the inline tags for emphasized, boldface,
and verbatim text. Plain subsitutions based on regular expressions
are used for this purpose.

The final step is to insert the code and TeX blocks again (these should
be untouched and are therefore left out of the previous parsing).

It is important to keep the Doconce format and parsing simple.  When a
new format is needed and this format is not obtained by a simple edit
of the definition of existing formats, it might be better to convert
the document to reST and then to XML, parse the XML and
write out in the new format.  When the Doconce format is not
sufficient to getting the layout you want, it is suggested to filter
the document to another, more complex format, say reST or
LaTeX, and work further on the document in this format.


===== A Glimpse of How to Write a New Translator =====

This is the HTML-specific part of the
source code of the HTML translator:
# #if FORMAT == "html"
(note that in HTML one of the the less-than and greater-than signs
in a link come up wrong because of the simple regex that is used
to substitute these pair of signs by special HTML expressions)
# #endif

# #if FORMAT != "epytext"

!bc
FILENAME_EXTENSION['html'] = '.html'  # output file extension
BLANKLINE['html'] = '<p>\n'           # blank input line => new paragraph
INLINE_TAGS_SUBST['html'] = {         # from inline tags to HTML tags
    # keep math as is:
    'math': None,  # indicates no substitution
    'emphasize':     r'\g<begin><em>\g<subst></em>\g<end>',
    'bold':          r'\g<begin><b>\g<subst></b>\g<end>',
    'verbatim':      r'\g<begin><tt>\g<subst></tt>\g<end>',
    'URL':           r'\g<begin><a href="\g<url>">\g<link></a>',
    'section':       r'<h1>\g<subst></h1>',
    'subsection':    r'<h3>\g<subst></h3>',
    'subsubsection': r'<h5>\g<subst></h5>',
    'paragraph':     r'<b>\g<subst></b>. ',
    'title':         r'<title>\g<subst></title>\n<center><h1>\g<subst></h1></center>',
    'date':          r'<center><h3>\g<subst></h3></center>',
    'author':        r'<center><h3>\g<subst></h3></center>',
    }

# how to replace code and latex blocks by html (<pre>) environment:
def html_code(filestr):
    c = re.compile(r'^!bc(.*?)\n', re.MULTILINE)
    filestr = c.sub(r'<!-- BEGIN VERBATIM BLOCK \g<1>-->\n<pre>\n', filestr)
    filestr = re.sub(r'!ec\n',
                     r'</pre>\n<! -- END VERBATIM BLOCK -->\n', filestr)
    c = re.compile(r'^!bt\n', re.MULTILINE)
    filestr = c.sub(r'<pre>\n', filestr)
    filestr = re.sub(r'!et\n', r'</pre>\n', filestr)
    return filestr
CODE['html'] = html_code

# how to typeset lists and their items in html:
LIST['html'] = {
    'itemize':
    {'begin': '\n<ul>\n', 'item': '<li>', 'end': '</ul>\n\n'},
    'enumerate':
    {'begin': '\n<ol>\n', 'item': '<li>', 'end': '</ol>\n\n'},
    'description':
    {'begin': '\n<dl>\n', 'item': '<dt>%s<dd>', 'end': '</dl>\n\n'},
    }

# how to type set description lists for function arguments, return
# values, and module/class variables:
ARGLIST['html'] = {
    'parameter': '<b>argument</b>',
    'keyword': '<b>keyword argument</b>',
    'return': '<b>return value(s)</b>',
    'instance variable': '<b>instance variable</b>',
    'class variable': '<b>class variable</b>',
    'module variable': '<b>module variable</b>',
    }

# document start:
INTRO['html'] = """
<html>
<body bgcolor="white">
"""
# document ending:
OUTRO['html'] = """
</body>
</html>
"""
!ec

# #else
Note that for Epytext, code or LaTeX blocks that contain a newline
character (for example as in `\nabla` in LaTeX), will lead to an
effect of the newline and generate error messages. Our remedy is
to remove such code blocks and provide a notice about the removal.
Eight here we only displacy a smaller snippet that Epytext can
treat properly:

!bc
INLINE_TAGS_SUBST['html'] = {         # from inline tags to html tags
    # keep math as is:
    'math': None,  # indicates no substitution
    'emphasize':     r'\g<begin><em>\g<subst></em>\g<end>',
    'bold':          r'\g<begin><b>\g<subst></b>\g<end>',
    'verbatim':      r'\g<begin><tt>\g<subst></tt>\g<end>',
    'URL':           r'\g<begin><a href="\g<url>">\g<link></a>',
    }
!ec

# #endif

===== Typesetting of Function Arguments, Return Values, and Variables =====

As part of comments (or doc strings) in computer code one often wishes
to explain what a function takes of arguments and what the return
values are. Similarly, it is desired to document class, instance, and
module variables.  Such arguments/variables can be typeset as
description lists of the form listed below and *placed at the end of
the doc string*. Note that `argument`, `keyword argument`, `return`,
`instance variable`, `class variable`, and `module variable` are the
only legal keywords (descriptions) for the description list in this
context.  If the output format is Epytext (Epydoc) or Sphinx, such lists of
arguments and variables are nicely formatted.

!bc
    - argument x: x value (float),
      which must be a positive number.
    - keyword argument tolerance: tolerance (float) for stopping
      the iterations.
    - return: the root of the equation (float), if found, otherwise None.
    - instance variable eta: surface elevation (array).
    - class variable items: the total number of MyClass objects (int).
    - module variable debug: True: debug mode is on; False: no debugging
      (bool variable).
!ec

The result depends on the output format: all formats except Epytext
and Sphinx just typeset the list as a list with keywords.

    - module variable x: x value (float),
      which must be a positive number.
    - module variable tolerance: tolerance (float) for stopping
      the iterations.

BIBFILE: manual_bib.bib, manual_bib.rst, manual_bib.py


