\input texinfo
@c %**start of header
@setfilename edb.info
@settitle EDB -- The Emacs Database
@paragraphindent 0
@c %**end of header

@c Copyright (C) 2004,2005,2006,2007,2008 Thien-Thi Nguyen

@dircategory Emacs
@direntry
* EDB: (edb).           The Emacs Database.
@end direntry

@copying
Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.

Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Free Software Foundation.
@end copying

@syncodeindex tp cp

@include version.texi

@titlepage
@title EDB Manual
@subtitle The Emacs Database

@author Michael Ernst
@author Thien-Thi Nguyen
@page
@vskip 0pt plus 1filll

Copyright @copyright{} 2004--2008 Thien-Thi Nguyen

@insertcopying
@end titlepage

@c ---------------------------------------------------------------------------
@node Top
@top EDB Manual

@ifinfo
EDB is the Emacs Database.  This file is the @value{UPDATED} Edition of the
EDB Manual, corresponding to EDB @value{VERSION}.

@quotation
@insertcopying
@end quotation
@end ifinfo

@menu
* Introduction::

User Manual
* Database mode::
* Database View mode::
* Database Edit mode::
* Searching::
* Sorting::
* Database Summary mode::
* Marking and hiding::
* Reports::
* Specifying the display format::

Programmer Manual
* Specifying control::
* Record field types::
* Database file layout::
* How information is displayed::
* Customization::
* Database representation::
* Naming conventions::
* In case of trouble::

Indices
* Function Index::
* Variable Index::
* Concept Index::

@detailmenu
 --- The Detailed Node Listing ---

Introduction

* Organization of this manual::
* Terminology and concepts::
* Invoking EDB::
* Example EDB session::

Database View mode

* Moving around in the database::
* Changing to Database Edit mode::
* Undoing all changes to a record::
* Making changes permanent::
* Adding and removing records::
* Exiting database mode::

Database Edit mode

* Exiting Database Edit mode::
* Undoing changes to a field::
* Moving from record to record::
* Moving from field to field::
* Movement within a field::
* Editing a field::
* Getting help::

Sorting

* Sort interface::
* Sorting functions::

Marking and hiding

* Setting the mark and hide bits::
* Movement among marked and hidden records::
* Details of hiding::

Specifying the display format

* Changing display formats::
* Execution of format file eval expressions::
* Making additional data display buffers::

Specifying control

* Interpreting control properties::
* Kinds of control property values::
* Control properties reference::
* New control example::
* Ongoing migration::
* Changing control properties at runtime::
* Example control file from scratch::

Record field types

* Specifying a record field type::
* Predefined record field types::
* Record field attributes::

Database file layout

* Data encoding::
* Internal file layout::
* Delimited file layout::
* Tagged file layout::
* Relational file layout::
* Nonregular file layout::
* Reading from disk::

Delimited file layout

* How to specify delimited file layouts::
* Resolving ambiguities::
* Problems with end-of-file newlines::

Nonregular file layout

* Nonregular database example::

How information is displayed

* Display specifications::
* Predefined displaytypes::
* Enumeration displaytypes::
* Defining new displaytypes::
* Display specification optional parameters::

Customization

* Auxiliary files::
* Hooks and customization functions::
* Global variables::

Hooks and customization functions

* Read hooks::
* Database mode hooks::
* Record display hooks::
* Edit mode hooks::
* Display format change hooks::

Database representation

* Mapping over the database::
* Manipulating records::
@end detailmenu
@end menu

@node Introduction
@chapter Introduction

EDB is a database program for GNU Emacs.  It permits you to manipulate
structured (or not-so-structured) data within Emacs and provides many of
the usual database features, including:

@itemize @bullet
@item
Flexible, customizable file layouts.  Data may contain any character,
including those used to delimit fields and records.  Files read and
written by the database may have arbitrary formats.

@item
Typed fields (e.g.@: integer, date, string); fields may also be subject
to additional constraints (prime number, date before today, string that
appears in some other record, etc.).

@item
Arbitrary data display formats for viewing records.  Multiple display
formats can be open on a database simultaneously, viewing the same or
different records.  The data display format can be automatically chosen
based on the record's field values.

@item
Selective display of only those records of interest; others become
temporarily invisible.

@item
Standard GNU Emacs editing commands, which work only within data fields
and not on the surrounding text.

@item
Database summaries, which show in a single buffer one or more lines of
information about each record.

@item
Sorting, with an easy-to-use interface for defining the
sorting criteria; most sorting orders you would care about are easy to
specify, but arbitrary ones are also permitted.

@item
Merging and reconciliation of databases.

@item
Reports generated from database information.

@item
Highly customizable via the underlying programming language, Emacs Lisp;
many hooks and useful variables are provided to make this even easier.
@end itemize

EDB is more ambitious --- and therefore more complex --- than its
forerunners (such as Forms Mode by Johan Vromans <jv@@mh.nl>).  While
other packages don't provide as much functionality as EDB, they may be
more appropriate for simple needs.

While EDB provides sophisticated functionality and is extensively
customizable, it is not as powerful as some other database systems,
and does not directly qualify as either ``relational'' or ``object-oriented.''
On the other hand, EDB may well meet your needs without being
buzzword-compliant.  (EDB extensions to provide a subset of the features of a
relational or object-oriented database can and have been written.)

@menu
* Organization of this manual::
* Terminology and concepts::
* Invoking EDB::
* Example EDB session::
@end menu


@node Organization of this manual
@section Organization of this manual

This manual contains two major parts.  The first part describes how to
use EDB to manipulate an existing database, and the second part
describes how to design a new database.

The first part --- which could be called the EDB User Manual --- first
presents basic commands such as loading a database into memory, adding,
deleting, and modifying records, and searching; then it describes
more advanced features such as sorting, displaying record
summaries, marking or ignoring certain records, and producing reports.

The second part --- which could be called the EDB Programmer
Manual --- describes the three forms that database information can take:
when being manipulated by EDB, when stored on disk, and when displayed
on the screen.  Separate chapters discuss specifying each of these
representations.  The manual then goes on to discuss customization hooks
and explains some of the lower-level implementation details that an
advanced programmer may need to know.

@cindex documentation to be deleted
@cindex implementation details
@cindex details, implementation
Note that these implementation details may change or disappear entirely, after
first becoming undocumented.  Such cases are marked with @kbd{Documentation
for FOO is to be deleted by EDB X.Y}, where X.Y is a version number of a
future EDB release.

@macro scadenza{v,x}
@emph{NOTE}: Documentation for \x\ is to be deleted by EDB \v\.
@end macro


@node Terminology and concepts
@section Terminology and concepts

A @dfn{database} is a collection of @dfn{records} kept in memory,
each of which is comprised of various @dfn{fields}.  A record's fields are
usually all related to some central object or concept; for instance, they
might describe various information about a particular person such as name,
address, and phone number.  A field's meta-information --- for example name,
default value, methods for representing the value between memory/disk/display
contexts, behavior on update, constraints, and so on --- is collectively
called its @dfn{fieldtype}.

All records of a database have the same structure (the number and order of the
fieldtypes are the same), though typically different records have different
information in the fields.

EDB permits database records to be viewed, edited, and manipulated in a
structured way.  When a database is @dfn{in memory}, there is at least one
@dfn{data display buffer} associated with it, showing a single record --- the
@dfn{current record} --- using a particular @dfn{display format}.  This buffer
can be in Database View or Database Edit mode.

A display format is a template, comprising @dfn{inter-field text} embedded
with at least one @dfn{display specification} (@dfn{displayspec}, for short).
The displayspec associates a field name with a @dfn{displaytype}, which
controls how a particular fieldtype is to be rendered in a buffer (inter-field
text is rendered as-is).

Each data display buffer may have a dependent @dfn{summary buffer} showing
selected fields of all records.  Moving from one record to another in the
summary buffer updates the data display buffer to synchronize the current
record.  (However, moving in the data display buffer does not update the
summary buffer.)

Killing a data display buffer kills its associated summary buffer (if any),
although the reverse is not true.  When the last data display buffer
associated with a database is killed, EDB forgets the database; it is no
longer in memory.

Collectively, all the information outside of the data proper --- display
formats, fieldtypes, and so on --- is called the @dfn{control}, to
borrow the hardware designer's terminology and awareness of the duality
of control signals versus datapath:

@quotation
If we believe in data structures, we must believe in independent
(hence simultaneous) processing.  For why else would we collect
items within a structure?  Why do we tolerate languages that give
us the one without the other?  ---Alan Perlis
@end quotation

[I believe Perlis is saying that the computation model for hardware,
where everything happens in parallel, is pretty interesting.  Maybe EDB
is not so interesting, but that's just a SMOP away...  ---ttn]


@node Invoking EDB
@section Invoking EDB
@cindex invoking EDB
@cindex files used by EDB

To use EDB, you need to establish a connection to a control source and a
data source, the former describing how EDB should handle the latter,
either with the command @code{edb-interact}, or with the
command @code{db-find-file}.

The first command made its debut in EDB
1.26 and is destined to evolve towards its final form for EDB 2.x; it is
recommended over the second one for reasons too complicated to explain
here (@pxref{Reading from disk}, instead).  Suffice to say that
the @code{db-find-file} approach will not be available in EDB 2.x.

In either case, to arrange for the commands to be autoloaded, add this
form to your @file{~/.emacs} file:

@lisp
(let ((el "database") (blurb "Emacs Database"))
  (autoload 'edb-interact el blurb t)
  (autoload 'db-find-file el blurb t))
@end lisp

Now, when you start up Emacs, you will already be able to execute
@code{edb-interact} and/or @code{db-find-file}; EDB will be
loaded automatically.

@subsection Connecting with @code{edb-interact}

@cindex @file{.edb} files
@cindex control file
@cindex control buffer
Control information is usually saved in a file, but may also be taken
directly from the current buffer.  This @dfn{control file} (or perhaps
@dfn{control buffer} as the case may be) is the first argument to the
command @code{edb-interact}.  The second argument specifies
the data file.

@deffn {Command} edb-interact control data
Open a connection to the @var{control} (.edb) and @var{data} files.
@var{control} may be a buffer whose contents are to be parsed as a
@file{.edb} file.  @var{data} may be nil, in which case the EDB searches
for a file whose stem is the same as @var{control}'s (or its
@code{buffer-file-name} if @var{control} is a buffer), with extension
one of @file{.data}, @file{.dat}, or the empty string (no extension), in
that order.

Once the connection is open, read in @var{data} and switch to the data
display buffer.  However, if @var{data} is already in memory, simply
switch to its (first) data display buffer.
@end deffn

You can use a properly constructed control file (or buffer) with
multiple data sources.  To create a new database, the easiest way is to
copy one of the @file{.edb} files in the distribution and adapt it to
your needs.  @xref{New control example}, for an example.

@subsection Connecting with @code{db-find-file}

@cindex @file{.dat} file suffix
@cindex @file{.fmt} file suffix
@cindex @file{.dba} file suffix
@cindex three file types used by EDB
You need three files to run EDB the ``old'' (1.x only) way:
a data file, a format file, and an
auxiliary file.  The data file (usual suffix @file{.dat}) contains the
information that makes up the database.  The format file (usual suffix
@file{.fmt}) specifies how the fields of a particular record appear in
the data display buffer, where you may view or edit one record at a
time.  The auxiliary file (usual suffix @file{.dba}) contains additional
information about the database, such as the number of fields in each
record, the layout of the data file (including what characters or
strings serve as field and record separators), customizations, etc.  A
fourth type of file is the report format file, which is a format file
used in generating reports printed on the screen, to a file, or to a
printer; @pxref{Reports}.
Different databases may share format and auxiliary files.

For examples of data, format, and auxiliary files,
see the @file{examples} subdirectory
of the EDB distribution.

You can combine the format and auxiliary files (by placing the
auxiliary file information at the end of the format file, in the local
variables section), but for simplicity we will consider the two files
separately.  There may be many different ways to lay out a record on the
screen, so a database could have many different format files; for the
time being we will concentrate on the format file which is used first,
which is called the primary format file, even though it might not be the
one that is used most often.

@cindex reading a database from disk
@cindex starting up EDB
When invoking the database, you typically only need to name the data
file; the names of the others are inferred from its name
(@pxref{Auxiliary files}) or may be mentioned explicitly by it.  The
@code{db-find-file} command loads the database into memory:

@deffn {Command} db-find-file [filename [promptp]]
Read a database from @var{filename}.
If the database file doesn't specify a format and the format file can't be
inferred from @var{filename}, then prompt for it, too.
Always prompt for the format file if prefix argument
@var{promptp} is non-@code{nil}.
If the database is already read in and @var{promptp} is @code{nil},
the existing database buffer is merely selected.
When called non-interactively, argument @var{promptp} may be a
string, the name of a format file to use.
@end deffn

@cindex saving to disk
@cindex writing to disk
You can use @code{db-save-database} and @code{db-write-database-file}
(normally bound to @kbd{C-x C-s} and @kbd{C-x C-w}, respectively, in Database
View mode) to save databases in memory to the file from
which they were read or to an arbitrary file.


@node Example EDB session
@section Example EDB session

Here is a small tutorial that steps through a typical EDB session: connecting
to a single control and a single data source, moving from record to record in
the data display buffer, creating a summary buffer, sorting the records,
writing the changed data (if any) to disk, and finally disconnecting.

Each step is a command or a series of commands to type into Emacs (sometimes
with trailing @key{RET} implied).  Additionally, you can type @kbd{C-h m} in
each of the buffers for more information.

@table @kbd
@item C-x C-f skram/retired.edb
@itemx M-x edb-interact
This visits a control file in the distribution and initiates a connection.
You can simply type @key{RET} when Emacs asks for the control file (at the
first prompt) to use the current buffer.  Note that you could have specified
the control file directly to @code{edb-interact} without visiting
it; we present it in two steps to give you a chance to look over its contents.

The distribution also includes the file @file{retired.data} in the same
directory, so you can type @key{RET} when Emacs asks for the data source (at
the second prompt) to have EDB find it automatically.

Once the data is in memory, Emacs switches to its data display buffer, which
is in Database View mode, which is read-only and does not permit editing.
@xref{Database mode}, @xref{Database View mode}.

@item n
@itemx p
These update the buffer with the next or previous record, respectively.
@xref{Moving around in the database}.

@item TAB
@itemx C-n
This moves point to the first field and switches to Database Edit mode.
@xref{Changing to Database Edit mode}, @xref{Database Edit mode}.

Once you are on a field, printing characters insert themselves and the other
usual editing commands work as well.  @kbd{TAB} moves to the next field, and
@kbd{C-n} moves to a field on the next line (or to the next line of this
field, if it spans multiple lines).
@xref{Moving from field to field}, @xref{Movement within a field}.

@item M-TAB
@itemx C-p
Like @kbd{TAB} and @kbd{C-n}, but move backward by fields or lines.

@item C-c C-c
This returns to Database View mode from Database Edit mode.
If you made any changes, you can type @key{RET} now to accept the changes,
or @kbd{C-x u} to undo them.

@item h
This displays a summary (also known as @dfn{headers}) of all the records of
the database.  You can move around in the summary
buffer using ordinary movement commands, and the record under point will
be displayed in the data display buffer.  Use @kbd{v} or @kbd{e} to
return to the data display buffer; @kbd{v} puts you in Database View
mode and @kbd{e} puts you in Database Edit mode.
@xref{Database Summary mode}.

@item S
(Note: uppercase.)  From Database View mode, this invokes
the database sort interface, which permits easy specification of
how records should be sorted.  Type @kbd{M-n} and @kbd{M-p} to drag
a field up (more important) or down (less important).  Type @key{RET}
to perform the sort and return to the data display buffer.
@xref{Sorting}.

@item C-x C-s
This saves to disk any changes you have made to the data.

@item +
This creates a new data display buffer.

@item x
This kills the current data display or summary buffer.
In the latter case, Emacs switches to the associated data display buffer.
You are offered the chance to save the changes you have made.
@end table

For invocation with @code{db-find-file}, the tutorial is still applicable if
you use @kbd{M-x db-find-file examples/forms-demo.dat} for the first step
instead of the @kbd{C-x C-f} and @kbd{M-x edb-interact}.


@node Database mode
@chapter Database mode

A single database record (typically the @dfn{current record}) is viewed in
a data display buffer.@footnote{The data display buffer was previously
called the format buffer; this is why all of the variables and functions
relating to it start with the @code{dbf-} prefix.}  The layout and
formatting of the data display buffer --- where and how the fields of the
current record are shown, and what fixed explanatory text surrounds
them --- is specified by a data display format.  Only the database fields
can be edited; the explanatory text is fixed.

@xref{Specifying the display format}, to create a new data
display format.
@xref{Making additional data display buffers}, to create a new data
display buffer (with the same or a different data display format).
@xref{Database Summary mode}, to view summary information about all
database records at once.

@cindex editing database files
@cindex database files, editing
@cindex files, editing database
Database mode has two basic submodes, Database View mode
(@pxref{Database View mode}) and Database Edit mode (@pxref{Database
Edit mode}).  These modes are used, respectively, when examining or
manipulating records and when changing information in a particular
record.  Keystrokes have different meanings in these two modes.  In
Database View mode no editing may be done, and many printable characters
are redefined to make manipulation of the database easy (for instance,
@kbd{n} moves to the next record).  In Database Edit mode a field of the
current record is being edited; most printable keys insert themselves,
and other editing and movement commands work in the ordinary way.  In
the data display buffer, where database records are ordinarily viewed
and edited, one of these two modes is always in effect.  (You may be
tempted to directly edit a raw database file in its on-disk layout.  Do
so at your own risk, and never change a database buffer out of Database
mode.)


@cindex mode line
The mode line indicates which mode you are in.  It looks something like:

@example
-***- machine-dbase        (Database Edit Abbrev 42/431)---All------------
@end example

The mode line consists of three modification indicators, the
name of the database file being manipulated (in this case,
@file{machine-dbase}), mode information within parentheses, and
the usual percentage-of-screen-visible indicator.  The mode
information consists of the database submode (such as View, Edit, or
Summary), any minor modes which are turned on (such as Abbrev
mode), the number of the current record, and the total number of records
in the database.

Ordinarily the Emacs mode line contains only one modification indicator
consisting of two dashes (not modified), asterisks (modified), or
percent signs (read-only).  The EDB mode line contains three
modification indicators, one each for the database, the displayed
record, and the current field.  The field indicator is @samp{*} if the
field under point has been modified, @samp{-} if it has not, and
@samp{%} if it is read-only or if no field is under point (for instance,
if the data display buffer is in Database View mode rather than Database
Edit mode).

The database is modified only when a changed record is written into it;
changes to the current record do not
immediately affect the database proper.  This permits such modifications
to be conveniently undone.  @xref{Undoing all changes to a record}.
@xref{Making changes permanent}.

Thus, the displayed record may be
modified without the database being modified, since the database is
considered modified only when the current record has been processed and
the resulting value placed in the database.  A similar situation exists
for the current field of the current record.  (@xref{Undoing changes
to a field}.)

Do not attempt to directly change the major mode of a database buffer.
If a database buffer is placed in another mode, the database functions
will cease working (they refuse to operate on non-database buffers,
since the consequences of such action could be severe); for instance,
you may be unable to save any of your work due to errors raised while
EDB tries write to its buffers to disk.  Furthermore, EDB makes
assumptions about where point is located in Database View and Database
Edit modes; violating these can cause changes to the current record to
be lost.


@node Database View mode
@chapter Database View mode
@cindex Database View mode
@cindex View mode, Database


The data display buffer is in Database View mode whenever field information is
not being edited.  Most commands to move from record to record and to
manipulate records (sorting, printing reports, showing summaries, etc.)
are performed in Database View mode.

Basic operations are described here; more complicated ones, such as
searching (@pxref{Searching}), are given sections of their own.

@menu
* Moving around in the database::
* Changing to Database Edit mode::
* Undoing all changes to a record::
* Making changes permanent::
* Adding and removing records::
* Exiting database mode::
@end menu


@node Moving around in the database
@section Moving around in the database
@cindex moving from record to record

@table @kbd
@item n
@deffn {Command} db-next-record [arg]
Go to the @var{arg}th next record.
In that record, go to the current field, if any.
@end deffn

@item p
@deffn {Command} db-previous-record [arg]
Go to the @var{arg}th previous record.
In that record, go to the current field, if any.
@end deffn

@item <
@itemx M-<
@deffn {Command} db-first-record [arg]
Show the database's first record.
With optional prefix argument, ignore hiding.
@end deffn

@item >
@itemx M->
@deffn {Command} db-last-record [arg]
Show the database's last record.
With optional prefix argument, ignore hiding.
@end deffn

@item j
@deffn {Command} db-jump-to-record arg [respect-hiding]
Show the database's @var{arg}th record.
Hiding is ignored unless optional argument @var{respect-hiding} is specified.
@end deffn
@end table

There are two special hybrid commands that show more of the current record
if there's more to see and otherwise show the next (or previous) record.

@table @kbd
@item @key{SPC}
@deffn {Command} db-next-screen-or-record [arg]
Go to the @var{arg}th next screenful of this display, or to the @var{arg}th
next record, if this is the last screenful of this display.
If point is in the summary buffer and the data display buffer is not visible,
then move to the next record.
@end deffn

@item @key{DEL}
@deffn {Command} db-previous-screen-or-record [arg]
Go to the @var{arg}th previous screenful of this display,
or to the @var{arg}th
previous record, if this is the first screenful of this display.
If point is in the summary buffer and the data display buffer is not visible,
then move to the previous record.
@end deffn
@end table


@node Changing to Database Edit mode
@section Changing to Database Edit mode

When in Database View mode, you cannot edit the fields of the record
being displayed.  In order to do so, change to Database Edit mode and
move to the field you wish to edit.  You can click the mouse on the
field you wish to edit, or move to the first or last field (and from
there to the desired field) via the following keystrokes:

@table @kbd
@item TAB
@itemx C-n
@itemx e
@deffn {Command} db-first-field
Move to first field.
@end deffn

@item C-p
@itemx M-TAB
@deffn {Command} db-last-field
Move to last field.
@end deffn
@end table


@node Undoing all changes to a record
@section Undoing all changes to a record
@cindex undoing changes to a record
@cindex reverting changes to a record

@table @kbd
@item C-x u
@deffn {Command} db-revert-record
Set the record to be the same as the corresponding one in the database.
In other words, undo any changes made since entering this record.
@end deffn

@item C-x r
@deffn {Command} db-revert-database
Replace the database in memory with the data on disk.
This undoes all changes since the database was last saved.
@end deffn
@end table

You can also undo changes to the current field; @xref{Undoing
changes to a field}.


@node Making changes permanent
@section Making changes permanent
@cindex making changes permanent
@cindex changes, making them permanent
@cindex committing changes
@cindex changes, committing them
@cindex accepting changes
@cindex changes, accepting them

You edit a copy of a database record; the database itself is not
changed until you commit the changes.  This occurs automatically
whenever any command causes a different record to be displayed, when the
database is saved, when a report is generated, and so forth.  It does
@emph{not} occur when you switch from Database Edit mode
to Database View mode,
though the field modification flag (in the mode line) will become a
percent sign and the record modification flag, an asterisk.  When the
record is committed, the record modification flag becomes a dash and
the database modification flag becomes an asterisk.

You can manually install the current record, as modified, into the
database.

@table @kbd
@item @key{RET}
@deffn {Command} db-accept-record
Install the current record in the database; make any changes permanent.
@end deffn
@end table

Committing a record makes the changes permanent only insofar as they
become part of the in-memory representation of the database.  The
on-disk version is not affected unless you overwrite it by using
@code{save-buffer} or @code{write-file}, or otherwise indicate that the
database should be written to disk (say, by responding affirmatively to
a question about saving the database).


@node Adding and removing records
@section Adding and removing records
@cindex adding a record
@cindex inserting a record
@cindex deleting a record
@cindex removing a record
@cindex killing a record

@table @kbd
@item a
@itemx i
@deffn {Command} db-add-record [append]
Add a new record to the database immediately before the current record.
Prefix arg @var{append} means to add after the last record, instead.  After
adding, move point to the new record's first field and switch to Database Edit
mode.
@end deffn

@item c
@deffn {Command} db-copy-record [arg]
Insert a copy of the current record in the database immediately after it.
The second of the two records is made the current record.
With a prefix argument, inserts that many copies.
@end deffn

@item o
@deffn {Command} db-output-record-to-db database
Copy (output) the current record to @var{database}.
@var{database} must be read in and compatible with the current database.
@end deffn

@item d
@itemx k
@deffn {Command} db-delete-record [arg]
Remove the current record from the database.
With a prefix argument, do not verify.
@end deffn
@end table


@node Exiting database mode
@section Exiting database mode
@cindex exiting database mode
@cindex quitting database mode
@cindex killing a database buffer

@table @kbd
@item q
@deffn {Command} db-quit
Quit editing the database for now; bury its buffers.
@end deffn

@item x
@deffn {Command} db-kill-buffer [noask]
Kill the data display buffer and any associated summary buffer.
Offer to save any changes.  Prefix arg means don't offer.
@end deffn

@item X
@deffn {Command} db-kill-all-buffers [noask]
Kill all buffers associated with the current database.
Offer to save any changes.  Prefix arg means don't offer.
@end deffn
@end table

For backward compatability, there is also the command @code{db-exit}.

@deffn {Command} db-exit [kill]
Be done with the database; like @code{db-quit}, but offer to save any changes.
With prefix argument, call @code{db-kill-all-buffers} after saving changes.
@end deffn


@node Database Edit mode
@chapter Database Edit mode
@cindex Database Edit mode
@cindex edit mode


The database is @dfn{modifiable} if either it is a new database, not
associated with any file, or if it is read from a file that is modifiable
(i.e., satisfies @code{file-writable-p}).  The database being not modifiable
does not prevent you from entering Database Edit mode, only from making
changes while in Database Edit mode.  You can toggle or set this status bit
with @code{db-toggle-modifiable-p}, normally bound to @kbd{C-x C-q} in
Database View, Database Edit, and Database Summary modes.

@deffn {Command} db-toggle-modifiable-p [arg]
Toggle whether the database may be modified.
With a nonzero prefix argument, set it modifiable.
With a zero prefix argument, set it non-modifiable.
@end deffn

The data display buffer is in Database Edit mode whenever a field of the
current record is being edited.  All printing characters insert
themselves, and the usual editing commands can be performed on the field
contents.  It is also possible to move from field to field and from
record to record, and to perform other operations on the database,
though the commands available in Database View mode are richer.

In Database Edit mode, point is always in the field currently being
edited.  The database is not modified as soon as changes are made in
Database Edit mode.  Rather, a copy of the record in question is
displayed and edited, and only when you move to a new record, initiate
some other global action (not specific to the edited record), or
explicitly commit the changes (@pxref{Making changes permanent}).  This
permits easier undoing of incorrect modifications.

In order to perform most record-level operations, you switch to Database
View mode, and then perform them there.  Several commonly used commands,
however, such as searching and moving from record to record, are
accessible directly from Database Edit mode.

Commands that move from field to field check the validity of the current
field before moving off it; commands that move from record to record do
this as well, then make any changes in the current record permanent (though
the database file on disk is not changed).

Basic operations are described here; more complicated ones are given
sections of their own.


@menu
* Exiting Database Edit mode::
* Undoing changes to a field::
* Moving from record to record::
* Moving from field to field::
* Movement within a field::
* Editing a field::
* Getting help::
@end menu


@node Exiting Database Edit mode
@section Exiting Database Edit mode

@table @kbd
@item C-c C-c
@deffn {Command} db-view-mode [arg]
Switch to Database View mode.
With an argument, toggle between Database View and Database Edit modes.
@end deffn
@end table


@node Undoing changes to a field
@section Undoing changes to a field
@cindex undoing changes to a field
@cindex reverting changes to a field

You can undo changes to the current field via Emacs' usual undo
facility; use @kbd{C-x u} or @kbd{C-_} to undo changes made since
entering the current field.

You can also revert the current field to its original value; this is
useful if you made a change, moved off the field, and then moved back
onto it.

@table @kbd
@item C-x U
@deffn {Command} db-revert-field
Undo any changes made since entering this field.
Replace the onscreen text in this field with that of the underlying record.

A similar effect can be had by invoking @kbd{C-x u} multiple times.
@end deffn
@end table

From Database View mode, you can simultaneously revert every modified
field of a record to its original value; @xref{Undoing all changes to a
record}.


@node Moving from record to record
@section Moving from record to record

The commands for moving to the next and previous records make any changes to
the current record permanent.  @xref{Moving around in the database}.


@node Moving from field to field
@section Moving from field to field
@cindex moving from field to field

@table @kbd
@item TAB
@deffn {Command} db-next-field [arg]
Move to @var{arg}th next reachable field, wrapping if necessary.
When called interactively, @var{arg} defaults to 1.
@end deffn

@item M-TAB
@deffn {Command} db-previous-field
Move to @var{arg}th previous reachable field, wrapping if necessary.
When called interactively, @var{arg} defaults to 1.
@end deffn

@item M-<
@itemx M->
Move to the first or last field (@pxref{Changing to Database Edit mode}).
@end table

Also see the keystrokes @kbd{C-n} and @kbd{C-p}, described below.


@node Movement within a field
@section Movement within a field

All Emacs cursor motion commands retain their standard meanings, except
that they do not move outside the field.

The line-movement commands have slightly changed meanings: if the motion
would take the cursor out of the current field, then they move to the next
field.

@table @kbd
@item C-n
@deffn {Command} db-next-line-or-field [arg]
Move to @var{arg}th next line.  If that would
move out of the current field, move to the closest field to that, but not the
current one, wrapping if necessary.
@end deffn

@item C-p
@deffn {Command} db-previous-line-or-field [arg]
Move to @var{arg}th previous line.  If that
would move out of the current field, move to the closest field to that, but
not the current one, wrapping if necessary.
@end deffn
@end table


@node Editing a field
@section Editing a field

Many Emacs editing commands retain their standard meanings; for
instance, printing characters insert themselves and deletion commands
work as usual, except that they do not make changes outside the field;
among these are:

@table @kbd
@item C-d
@code{delete-char}
@item @key{DEL}
@code{backward-delete-char}
@item M-d
@code{kill-word}
@item M-@key{DEL}
@code{backward-kill-word}
@item C-k
@code{kill-line}
@end table


@node Getting help
@section Getting help
@cindex help for record fields

You can get some information about the current field, such as what type of
value it expects or what its contents signify, by using the following
command.

@table @kbd
@item M-?
@deffn {Command} db-field-help
Display help for current field in the echo area.
@end deffn
@end table


@node Searching
@chapter Searching
@cindex searching

A useful and commonly used database operation is searching for records that
meet some criteria: for instance, finding a particular record or indicating
that following operations should only apply to records that correspond to
an address in greater Boston.  EDB provides several functions to support
such operations.

@section Search commands
@cindex search commands

To perform a search pertaining to the contents of only one field, move to
that field and use the following command:

@table @kbd
@item M-s
@deffn {Command} db-search-field
Search for occurrences of @var{pattern} in the current field of any record.
Finds the first match after the current record; wraps around automatically.
With prefix argument, marks all matches in addition to going to the first one.
If hiding is in effect, hidden records are ignored.
@end deffn
@end table

The same keystroke in Database View mode permits specification of patterns
which depend upon the contents of several fields:

@table @kbd
@item s
@itemx M-s
@itemx M-S
@deffn {Command} db-search
@code{db-search} is not yet implemented; use @code{db-search-field} instead.
In a future version of EDB, @code{db-search} will permit searching on all
fields of a record simultaneously.
@end deffn
@end table

For a description of marking, @xref{Marking and hiding}.

@section Search patterns
@cindex search patterns

Search patterns can be as simple as a datum to match exactly or as
complicated as the conjunction, disjunction, and negation of tests to be
performed on field contents.

@subsection Basic patterns

A basic search pattern has the same form as the data that is kept in the
field; for instance, to search in a string field for a particular
string, use that string; to search for the date March 14, 1967, use
@samp{3/14/67} or @samp{14 March 1967} or any other accepted date
format.  A basic pattern is treated somewhat more richly than a literal,
however.  In a string field, typing a string results in a match for any
element which contains it as a substring; typing @samp{ail}, without the
quotes of course, matches both @code{"ailment"} and @code{"fail"}.
In a date
field, @samp{3/67} matches any date in March of 67, not just those
March 67 dates which specifically exclude a day of the month.

@subsection Comparisons

@cindex <, in search pattern
@cindex >, in search pattern
@cindex =, in search pattern
A search pattern may also be a comparison prefix (<, >, or =) plus a
datum which is treated exactly like any other element of that field.  In
a string field, @samp{<ail} matches all elements lexicographically less
than @code{"ail"}; @samp{=ail} matches only fields containing exactly
@code{"ail"}, but not @code{"ailment"} or @code{"fail"}.
Warning: @samp{= ail} looks
for an entry containing the four-character sequence
starting with a space: @w{@code{" ail"}}.

In a date field, searches work slightly differently; for instance,
@samp{>3/14} matches dates after March 14 in any year, and
@samp{=3/67} matches only dates in March 67 whose day of month is not
specified.  For more information about the interpretation of patterns,
see the documentation for the particular types.

@subsection Logical connectives

More complicated patterns can be built up out of simpler ones via the
logical connectives AND, OR, and NOT.  These work in the obvious way.
One pattern which finds any date between the ides of March and
Christmas, inclusive, is @samp{> 3/14 AND < 12/26}; two patterns which
find dates except March 14 are @samp{NOT 3/14} and @samp{< 3/14 OR >
3/14}.  To find strings that either contain the substring @code{"ail"} or
start with a, b, or c, use @samp{ail OR < d}.

The precedence of these connectives is: NOT, which is most tightly bound
to its test, then OR, then AND.  There are no provisions for grouping or
otherwise overriding this ordering.  Connectives (and REGEXP,
described below) consume all surrounding spaces and tabs.


@subsection Other pattern operations

One other pattern operation of interest is the regexp operator for
string fields.  This is invoked by either using REGEXP with surrounding
spaces, or / without a trailing space.  For instance, @samp{/^[ace]}
matches any string field starting with a, c, or e; @samp{/.} matches any
nonblank string field; and @samp{NOT REGEXP a.*b.*c} matches any string
which does not contain the letters a, b, and c in order.  This last
example shows that EDB's search commands are more powerful than general
regular expression searching.


@node Sorting
@chapter Sorting
@cindex sorting

Sorting a database reorders its records without changing the contents of
any particular record.  For instance, it might be convenient to arrange
an address database sometimes alphabetically by last name, sometimes by
ZIP code, and sometimes by some other criteria.

@menu
* Sort interface::
* Sorting functions::
@end menu


@node Sort interface
@section Sort interface

From a data display buffer in Database View mode, you can specify a
particular record ordering using @code{db-sort}.

@table @kbd
@item S
@deffn {Command} db-sort [arg]
Switch to the sort interface buffer.
With a prefix argument, don't confirm the sort order.
@end deffn
@end table

Ordinarily, calling this function invokes the sort interface, an
easy-to-use tool for specifying which fields should be treated as sort
keys, and in what order.  In a database whose records have fields
@samp{firstname}, @samp{lastname}, @samp{address}, @samp{city}, and
@samp{zip}, and in which records should first be sorted on
@samp{lastname} in increasing order, then on @samp{firstname} in
@emph{decreasing} order, ignoring the other fields entirely for the
purposes of the sort, the sort interface display would look like

@example
==== Significant fields:
  lastname            increasing
  firstname           decreasing
==== Nonsignificant fields:
  address             increasing
  city                increasing
  zip                 increasing
==== Hidden records to end:  No
@end example

In addition to which fields should be sorted by, the sort interface
permits specification of how hidden records should be treated.

@table @kbd
@item t
@deffn {Command} dbsi-toggle-hidden-to-end
Toggle a flag that
controls whether hidden records should all be placed at the end
of the sorted order or should be sorted according to the same criteria as
non-hidden records.
@end deffn
@end table

To change the relative order of fields, and whether they're significant or
not, use the following commands.

@table @kbd
@item C-k
@deffn {Command} dbsi-kill-line
Move field on current line to the top of the nonsignificant list.
This can also be used within the nonsignificant list to reorder it.
@end deffn

@item C-y
@deffn {Command} dbsi-yank-line
Yank the field from the top of the nonsignificant list.
Insert it before the current line.  Prefix arg means point doesn't move.
@end deffn

@item M-p
@itemx M-n
In the list of significant fields, ``drag'' the field under point
up (more important) and down (less important), respectively.
@end table

To specify how a particular field should be ordered, use the following
commands.

@table @kbd
@item i
@deffn {Command} dbsi-increasing
Specify that the field at point should use an increasing ordering.
@end deffn

@item d
@deffn {Command} dbsi-decreasing
Specify that the field at point should use a decreasing ordering.
@end deffn

@item o
@deffn {Command} dbsi-ordering-function
Specify an ordering function for the field at point.
An @dfn{ordering function} returns -1, 0, or 1 depending on whether its first
argument is less than, equivalent to, or greater than its second argument.
@end deffn

@item s
@deffn {Command} dbsi-sorting-function
Specify a sorting function for the field at point.
A @dfn{sorting function} returns @code{t} if its first argument is less
than its second argument and @code{nil} otherwise.
@end deffn
@end table

Each database has a default sort order specified by the @code{:field-order}
control property (@pxref{Control properties reference}),
used when setting up the
sort interface (and is used for sorting when no other ordering
information is specified).  When you exit the sort interface,
the database's default ordering
can be set to the ordering depicted on the screen (see below).

Each data display buffer also has a default sort order; the database's
default order is ignored if a local value is chosen
using the sort interface.
Setting the database's sort order automatically clears the local value.

The following commands are used to exit the sort interface; most of them
also cause the database to be sorted.  Some of them set the default
ordering for the database or the data display buffer.

@table @kbd
@item @key{RET}
@itemx C-c C-c
@deffn {Command} dbsi-use-ordering-make-database-default
Use the current ordering to sort, and make it the default for future sorts
of this database.
Display a warning if there are killed, non-yanked fields.
@end deffn

@item A
@itemx U
@deffn {Command} dbsi-use-ordering-make-buffer-default
Use the current ordering to sort, and make it the default for future sorts
in this data display buffer only.
Display a warning if there are killed, non-yanked fields.
@end deffn

@item a
@itemx u
@deffn {Command} dbsi-use-ordering
Use the current ordering for this sort only.
@end deffn

@item !
@deffn {Command} dbsi-this-field-only
Sort according to only the field at point.
All editing of other fields is ignored.
@end deffn

@item q
@deffn {Command} dbsi-quit
Abort the sort and exit the sort interface
(@code{dbsi-quit}).
@end deffn

@item c
@deffn {Command} dbsi-quit-clear-buffer-default
Clear the default sort order for this buffer and exit the sort interface
without sorting.
In the future, the default sort order will come from the database.
@end deffn
@end table

The sort interface returns a field priorities list to be used when
sorting.

Sorting does not ordinarily mark the database as modified, because not
the data itself, but only the way it is arranged, has been changed.  If
you set @code{db-sort-modifies-p} to non-@code{nil}, then whenever a
database is sorted (even if the resulting order is the same as the
original one), the database is marked as modified.

@defvar db-sort-modifies-p
If non-@code{nil}, then sorting a database marks it as modified too.
@end defvar

When using control properties, you should use a @code{:locals} variable
by the same name.

@node Sorting functions
@section Sorting and ordering functions
@cindex sorting functions
@cindex ordering functions

In order to specify the relative order of two field values (for the same
field, but from different records), you provide a
sorting function, an ordering function, or both.  If only one is
provided, the other is automatically generated from it.  In any event,
only one of them is used for a given field on any particular sort.

A sorting function takes two field values as arguments and returns
@code{t} if its first argument is less than its second argument (that
is, the first argument appears previous to the second in the sorted
order).  The sorting function returns @code{nil} if the arguments are
equal or if the first argument is greater than the second (appears later
in the sorted order).

An ordering function, on the other hand, returns complete information
about the relative order of its two arguments: it returns -1, 0, or 1
depending on whether its first argument is less than, equivalent to, or
greater than its second argument.

Use of an ordering function can result in fewer comparisons in some
cases, because it returns more information.  This is worthwhile if a
significant amount of processing is required before the comparison is
done.  For example, suppose that two addresses are being carefully
checked for equality and some of the steps leading up to that are
expansion of abbreviations, standardization of spelling, capitalization,
and spacing, etc.; then it is better to return the exact relative
ordering than to possibly require another time-consuming operation to
determine it.

If it is possible to canonicalize the values beforehand, that may be
even more efficient, but that is not always possible; consider the case
of a small (but tedious to extract) part of the information in each
field being compared.


@node Database Summary mode
@chapter Database Summary mode
@cindex Database Summary mode
@cindex summary mode

A summary is a listing containing abbreviated information about every
record; it permits many records to be viewed at once.

This is available in Database View mode via the following command:

@table @kbd
@item D
@itemx H
@itemx h
@deffn {Command} db-summary
Display a summary (or directory) of all database records according to
the value set by
@code{dbf-set-summary-format}.
The summary appears in a separate buffer.
When called from the summary buffer, this updates the summary.
@end deffn
@end table

When a summary is created, the summary display format appears in the
summary buffer once for each record, with appropriate values substituted
for its display specifications.

The entire format is indented by two characters; the first and second
columns contain @samp{+} and @samp{[}, respectively, if the record is
marked or hidden.
@xref{Marking and hiding}.  @xref{Details of hiding}.

The summary buffer is not updated whenever a record value changes; in
the interest of efficiency, it remains as is until the next
@code{db-summary} command is issued, at which time the summaries are
redisplayed after all, some, or none of them have been recomputed.  For
the same reason, when a single mark or hide bit changes, the summary is
updated; when many change, it is usually not.

When point is in the summary buffer, the associated data display buffer
(nearly) always displays the record under point.  (Some Emacs commands
can move point without EDB noticing.)  Movement in the summary buffer is
by any of the ordinary Emacs commands, including searching.  Most
Database View mode commands also work in the summary buffer.

The summary display format defaults to the first non-literal line in the
database display format --- that is, the first line which contains a display
specification.

@cindex summary format, setting
Summary display formats can display information any way that an ordinary
display format can, including showing more than one field of the record
in question or spanning several lines.  The only restriction is that the
summary display format must cover a specific number of lines: each
display specification must have its @code{min-height} and
@code{max-height} slots set to equal values.  For more information about
display formats, @xref{How information is displayed}.

The following command may appear in a format file or an auxiliary file,
or it can be invoked directly.  When it appears as an Emacs Lisp form,
remember the special meaning of the backslash character and double it
where necessary.

@deffn {Command} dbf-set-summary-format summary-format
Specify the format used in the Database Summary buffer.
Argument @var{summary-format} is a string containing display specifications.
Call this in the data display buffer, or in a format file or auxiliary file.
@end deffn


@node Marking and hiding
@chapter Marking and hiding
@cindex marking
@cindex hiding
@cindex selecting only some records

The marking facility permits operations to be performed on only certain
records of a database.  For instance, to create a report which describes
only some of the database records, you would first mark the records of
interest.  Then you would call @code{db-report} with a prefix argument (do
so by pressing @kbd{C-u} first), or would make optional second argument
@var{markedp} non-@code{nil}.  See the documentation of the individual
operations to see whether they support operation on only the marked
records.

The hiding facility also restricts attention to a subset of the current
database.  It is similar to Outline mode in that it makes some
records temporarily invisible.  Hiding is useful when you wish to
concentrate on a subset of the database without being distracted by
other records that may be present.  By default hidden records are
skipped by the record-motion commands, excluded from searches and
reports, and ignored by most other operations.

Mark and hide bits are associated only with the in-memory version of a
database; they are not saved when you save a database, and are always
unset (boolean false) when a database is read in.  (When a database is
written to disk, although the disk version of the data does not contain
the mark and hide information, that information is not lost from the
working copy of the database.)

This behavior is a feature, not a bug.  Marking and hiding are
intended to help you temporarily group database records for operations
upon them; if there is information that cannot be recreated from a
record's fields, then you should consider adding another field for that
information.  On the other hand, mark or hide criteria may be
complicated.  If such a pattern is used often, then you may wish to
write a function to set the bits appropriately, which function could be
bound to a keystroke or automatically executed when the database is read
in.


@menu
* Setting the mark and hide bits::
* Movement among marked and hidden records::
* Details of hiding::
@end menu


@node Setting the mark and hide bits
@section Setting the mark and hide bits

Every record may be thought of as having a pair of bits or boolean values
indicating whether it is marked and whether it is hidden.  The most
straightforward way to set these bits is to use an operation to mark,
unmark, hide, or unhide a particular record; these are bound to
keystrokes in Database View mode.

@table @kbd
@item m
@deffn {Command} db-mark-record
Toggle whether the current record is marked.
With a nonzero prefix argument, set it to be marked.
With a zero prefix argument, set it to be unmarked.
@end deffn

@item O
@deffn {Command} db-hide-record
Change whether the current record is hidden.
With a nonzero prefix argument, set it to be hidden.
With a zero prefix argument, set it to be unhidden.
@end deffn
@end table

The searching commands, when called with a prefix argument, mark each
matching record (@pxref{Searching}).

Once all records of interest have been marked, through one or more marking
and/or searching commands, unmarked records can be hidden from
consideration.  This is useful if you want to work on only a small number
of records, or if specifying the records of interest is easier than
specifying those not of interest: instead of hiding all the
uninteresting records, simply mark the interesting ones, then use the
following command to cause the unmarked ones to become hidden.

@deffn {Command} db-hide-unmarked-records
Enable hiding, hide all unmarked records, and clear all mark bits.
@end deffn

The converse operation transfers information from the hide bits to the mark
bits.

@deffn {Command} db-mark-unhidden-records
Mark all unhidden records.  Also clears all hide bits.
@end deffn

It is also possible to clear all the mark or hide bits.

@deffn {Command} db-unmark-all
Clear the mark bit of every record.
@end deffn

@deffn {Command} db-unhide-all
Clear the hide bit of every record.
@end deffn

Generally, to set or clear either the mark or hide bits of an arbitrary
record of a database, you need to obtain the appropriate @dfn{tag}
with @code{edb-tag}, passing it and the record to one of the functions
described below.

@defun edb-tag name db
Return the tag object named @var{name} associated with database @var{db}.
For EDB 1.x, @var{name} can be one of the keywords: @code{:markedp} or
@code{:hiddenp}.  The tag object can be passed to @code{edb-tagp},
@code{edb-tagx} and @code{edb-tag-}.
@end defun

@defun edb-tagp tag record
Return t if @var{record} has its @var{tag} set.
@end defun

@defun edb-tagx tag record
Set @var{tag} for @var{record}.
@end defun

@defun edb-tag- tag record
Clear @var{tag} for @var{record}.
@end defun

For example, this function sets the mark bit on those records
(@pxref{Mapping over the database}) for which @var{predicate} returns non-nil:

@lisp
(defun my-mark-if (predicate)
  (let ((mtag (edb-tag :markedp dbc-database)))
    (db-maprecords (lambda (record)
                     (when (funcall predicate record)
                       (edb-tagx mtag record))))))
@end lisp


@node Movement among marked and hidden records
@section Movement among marked and hidden records

Ordinarily, record movement commands (those which move from one record to
another) ignore hidden records, so that you never land on a hidden
record.  Marked records, on the other hand, are not treated specially by
the record movement commands.  The following Database View mode keystrokes
permit you to move to hidden records or to move directly to marked
records.

@table @kbd
@item M-n
@deffn {Command} db-next-record-ignore-hiding [arg]
Go to the @var{arg}th next record, ignoring omissions.
That is, all records, even those which are hidden, are counted.
@end deffn

@item M-p
@deffn {Command} db-previous-record-ignore-hiding [arg]
Go to the @var{arg}th previous record, ignoring omissions.
That is, all records, even those which are hidden, are counted.
@end deffn

@item M-C-n
@deffn {Command} db-next-marked-record [arg]
Go to the @var{arg}th next marked record.
Hidden records are treated according to db-hide-p.
@end deffn

@item M-C-p
@deffn {Command} db-previous-marked-record [arg]
Go to the @var{arg}th previous marked record.
Hidden records are treated according to db-hide-p.
@end deffn
@end table



@node Details of hiding
@section Details of hiding

The @dfn{hidden} bit of each record has no effect unless
@dfn{hiding is in effect}.

When hiding is not in effect, the values of records' hide bits
are remembered, and they may still be set and unset, but they have no
effect on any operations until hiding is once again in effect.
Enabling hiding makes @samp{Hide} appear in the mode line of
the database buffer.

The following operations work in either Database View mode or Database
Summary mode.

@table @kbd
@item M-o
@deffn {Command} db-hiding-toggle [arg]
Change whether hiding is in effect.
With a nonzero prefix argument, turn hiding on.
With a zero prefix argument, turn hiding off.

This does not change the current hide-function, and a hide bit is always
computed for each record, but hide bits have no effect on any operations
if hiding is not in effect.
@end deffn

@item M-C-o
@deffn {Command} db-toggle-show-hidden-records [arg]
Toggle whether hidden records are shown in the summary.
With a nonzero prefix argument, show hidden records in the summary.
With a zero prefix argument, don't show hidden records in the summary.
@end deffn

@item M-O
@deffn {Command} db-hiding-set
Set the criteria for automatically determining whether to hide a record.
This isn't implemented yet.
@end deffn
@end table


@node Reports
@chapter Reports
@cindex reports

Reports can be generated from a database by using the following command
in view mode:

@table @kbd
@item r
@deffn {Command} db-report [report-filename [markedp]]
Create a report according to @var{report-filename}.
Prefix argument @var{markedp}, if non-@code{nil}, means report on only
marked records.
If hiding is in effect, hidden records are not reported upon.
When called interactively, prompts for @var{report-filename}.
@end deffn
@end table

The way a report looks is specified in precisely the same as are display
formats and summary formats (@pxref{How information is displayed}).
This information must be placed in a file; you cannot type it
directly when creating a report.  This restriction makes errors in the
report format easier to correct.

The report is placed in the @samp{*Database Report*} buffer, which is in
Text mode.  The information may then be edited, saved to disk, or
otherwise manipulated.  The buffer is in Text mode and is not yet a
file; you must save it to make it a file.

To create a report which mentions only marked records
(@pxref{Marking and hiding}), supply a prefix argument to the report
command by typing @kbd{C-u} first.


@node Specifying the display format
@chapter Specifying the display format
@cindex display format, selecting

Different layouts and on-screen arrangements of the values stored in
database records are appropriate when you are concentrating on different
aspects of the data.  Sometimes you prefer to see just a few of the
fields; at other times you may want to see the records in full detail.
It may also be appropriate for the display format of a record to depend
on the record's field values.  This section describes how to choose a
different display format for the record being displayed, either manually
or automatically.


@menu
* Changing display formats::
* Execution of format file eval expressions::
* Making additional data display buffers::
@end menu


@node Changing display formats
@section Changing display formats
@cindex changing display formats
@cindex alternate display formats
@cindex variant display formats
@cindex display format, alternate
@cindex display format, variant

EDB permits the creation and use of a variety of display formats with a
single database; you can also conveniently change the way that a
particular record is displayed by using @code{db-change-format} and
specifying the filename of the new display format, or a nickname for the
format that has been assigned either during the session or before-hand.
Choosing a different format does not create a new data display buffer;
it changes the way that records are displayed in the current one.

@deffn {Command} db-change-format [format-name [filename]]
Select and use an alternate display format to view the database.
If neither @var{format-name} nor @var{filename} is specified (as is the
case when this is called interactively), then prompt for them.
In Emacs Lisp
code, if @code{dbf-format-name-spec-alist} has been been set, usually
only one of the arguments is specified.  If both are specified, then
@var{format-name} becomes a name for the format @var{filename} specifies;
if @var{format-name} is already
associated with a different format file, an error is signalled.

If the current format is unnamed, then prompt for a name
to give it, so that it can be conveniently restored if need be.  This
behavior is suppressed, and the record is not displayed, if the function is
not being called interactively.

The data display buffer is left in Database View mode.

Selecting the current format does not cause any work to be done.

Some databases automatically set the format of the record being displayed,
usually by setting @code{dbf-before-display-record-function} to a function
that overrides the format in effect when a record is about to be displayed.
This may cause this function to appear not to be doing any work.  In
actuality the format is being set, then reset.
@end deffn

@cindex format-spec structure
@cindex specifier, format
@defvar dbf-format-name-spec-alist
Association list of format names and format specifiers.
Each format name is an arbitrary string.
A format specifier is a filename or format file specifier, which is
a list of values for format variables.
When the format specifier is a filename, after that format file
has been read, EDB replaces the filename with a list of values for format
variables, so that the file need not be read again.

This variable is set automatically for each @code{:display} control property
specified (@pxref{Control properties reference}).  If there is only one
@code{:display} and it is without a name, its name is taken as @file{t}.
@end defvar

These values can be set in the auxiliary or format files so that you can
choose a format name (with completion) instead of having to remember a
filename.  If the selected format's specifier is a filename, then after
the file is read in, the format-spec is modified by replacing the
filename with information about the format such as the displayspecs, the
invariant text between them, and so forth.  Subsequent selections of
that format do not cause disk accesses.  For an example of the use of
@code{db-change-format}, @xref{Record display hooks}.

@cindex hiding fields from a data display buffer
@cindex data display buffer, hiding fields
While it is not currently possible to selectively hide certain fields
from a data display buffer, judicious use of alternate formats can
result in nearly the same effect.


@node Execution of format file eval expressions
@section Execution of format file eval expressions

@cindex local variables section of format file
@cindex format file, local variables section
@cindex format file, eval expressions in
@cindex eval expressions, in format file
@cindex primary format file
@cindex format file, primary
Often the local variables section of a format file contains code
that should only be executed once, or should only be executed before the
database is read in, because of either efficiency or correctness
constraints.  Because of this, the local variables section of a
format file is executed only when it is read in from disk (which is
usually only once).  In order to cause an expression to be evaluated
every time that a particular display format is selected, use the
@code{dbf-always} macro:

@defmac dbf-always [body...]
Execute forms in @var{body}, and arrange to execute them in the future
each time that this format replaces another.
@end defmac

Of course, it is often valuable to overwrite a value when the display
format changes; this is the purpose of @code{dbf-always}.  It is always
safe to set variables whose name begins with @code{dbf} in such forms,
though changes to some such values will not take effect when a display format
is being returned
to (though they will work when it is first chosen).  This will affect
you only in that a call to @code{dbf-set-summary-format} will have an
effect only the first time that a format file is read in, not every time
that it replaces another, even if it is enclosed in a @code{dbf-always}
form.

The @code{dbf-always} forms are not executed every time that a
record is displayed, or even every time that @code{db-change-format}
is called, but only when a format replaces another one (that is,
@code{db-change-format} is called and its first argument is not equal
to the name of the current format).

Here is an example of a common problem with an expression which causes
an error if evaluated every time that the format is selected.  The
primary format file (the one that is used when the database is read in)
is permitted to define the fields (names and types) of the database;
other parts of the database initialization code propagate
that information into other parts of EDB, caching and transforming
things in the process.  If you
switch to another display format and back to the primary one, and the
the field names have been re-initialized to their original forms again,
then the next
attempt to access this information (expecting a pre-transformed object)
would cause an error.  In this case, the proper solution is to use
@code{database-set-fieldnames-to-list}
(@pxref{Specifying a record field type})
instead, but such functions are not provided for every slot that it
would be dangerous to set.  If several format files all set a value
which is dangerous to change, then another possibility is to check the
value before setting it: if it is already set, then don't do anything.
Another possibility is to move all assignments to database slot values
from the format file to the auxiliary file.


@node Making additional data display buffers
@section Making additional data display buffers
@cindex additional data display buffers, making
@cindex data display buffers, making additional
@cindex two records, manipulating simultaneously
@cindex simultaneously manipulating two records
@cindex two formats, using simultaneously
@cindex simultaneously using two formats

In addition to changing the display format of an existing data display
buffer, it is sometimes useful to have two different data display
buffers both examining the same database, either so that two different
records can be viewed or edited simultaneously or so that two different
formats can be used at the same time --- or both.

Use the following function (normally bound to @kbd{+} in Database View
mode) to create a second (or additional) data display buffer for the
current database.

@deffn {Command} db-additional-data-display-buffer
Create another data display buffer in which to view this database.
@end deffn

If you edit the same record in more than one data display buffer, only
the last one committed (by calling @code{db-accept-record}, moving to
another record, saving the database, etc.)@: has an effect.
(Simply switching from Database Edit mode to Database View mode does
@emph{not} commit the changes; failing to commit changes makes it
appear that changes in one data display buffer are not being
communicated to the other ones associated with the same database.  For
more on committing, @xref{Making changes permanent}.)  It is
perfectly safe, however, to edit different records of the same database
in different data display buffers, or to perform any other database
manipulations.

To learn about formatting directives and specification of display format
files, @xref{How information is displayed}.


@node Specifying control
@chapter Specifying control
@cindex specifying control
@cindex control properties

The control for a particular data source comprises several different types of
specification, describing all aspects of the data with respect to a connection
session: its shape in abstract, its internal relationships, its representation
outside of Emacs, its representation in an Emacs buffer, and various other
meta-data.

These are collectively known as the @dfn{control properties}, most of which
must be specified in the process of making a connection, by grouping them
into a control file (or buffer) that begins with the magic text:

@example
:EDB (single BASIS) ;;; -*- emacs-lisp -*-
@end example

Actually, the @kbd{;;; -*- emacs-lisp -*-} is optional; it is sufficient that
on the first non-comment, non-blank line,
the first four characters are colon and @kbd{EDB}, followed by a space
followed by a list whose first element is the name of a supported
@dfn{schema-schema}.

At the moment (for EDB 1.x), the unique available schema-schema, and the
subject of the rest of this manual, is @code{single}.  Also, not all control
can yet be specified with control properties; some ``normal'' Emacs Lisp
variables and functions are still required.  @xref{Ongoing migration}.

@var{basis} is an optional variable name (symbol) whose value specifies in
property list form a set of control properties that are to be used as the
basis --- i.e, elaborated (@pxref{Interpreting control properties}) first ---
for those specified in the rest of the buffer.  This is useful for collecting
in one place properties that are shared by many databases.

@menu
* Interpreting control properties::
* Kinds of control property values::
* Control properties reference::
* New control example::
* Ongoing migration::
* Changing control properties at runtime::
* Example control file from scratch::
@end menu


@node Interpreting control properties
@section Interpreting control properties
@cindex interpreting control properties
@cindex control properties, interpretation

The way EDB processes a control file is designed so that you can express
control properties primarily declaratively, while still permitting arbitrary
computation, all in one place.  Besides the magic header
(@pxref{Specifying control}), the rest of the control file contains, at the
top-level, a mix of free-standing Lisp, and alternating control property names
(keywords) and their associated @dfn{values-producing forms}.

Unlike @code{load} (or @code{load-file}), which can be described as a
straightforward read-eval loop, control file processing is a bit more
complicated.  Specifically, EDB:

@enumerate
@item Checks the magic.

@item Parses the buffer using @code{read} and some helper funcs,
accumulating the free-standing Lisp separately from the control properties.
The helper funcs transform the text blocks associated with certain control
properties into a plist.  For example, given a buffer fragment:

@example
:display t
first
second
:EOTB
@end example

@noindent
Then parsing yields the form (associated with @code{:display}):

@lisp
(:name t :coding t :EOTB "EOTB" :text "first\nsecond\n")
@end lisp

@item Evaluates, in order, the free-standing Lisp.

@item Validates the syntax of the control properties.

@item ``Elaborates'' the form associated with each control property,
to get the actual control property values, and superficially checks these
values.  The elaboration method depends on the form:

@itemize
@item A list whose @sc{car} is a string is used directly.
@item A list whose @sc{car} is a keyword is used directly.
@item For all other cases, @code{eval} the form.
@end itemize

This allows (certain) control property values to be computed at connection
time.  For example, here are two equivalent ways to set the @code{:fields}
control property:

@example
;;; first way: literal
:fields [time place]

;;; second way: computed
:fields (vector 'time 'place)
@end example

Here is another example which shows four functionally-equivalent ways to set
the @code{:write-record} control property:

@example
;;; first way: external function
(require 'my-edb-setup)
:write-record 'my-write-record

;;; second way: defun
(defun my-write-record (time place) ...)
:write-record 'my-write-record

;;; third way: defun integrated into decl
:write-record
(defun my-write-record (time place) ...)

;;; fourth way: lambda (the ultimate)
:write-record (lambda (time place) ...)
@end example

Although the ways are functionally equivalent, each has its consequences.
The first way allows you to use a byte-compiled function (presuming it is
defined in @file{my-edb-setup.el} and you byte-compile that file).
Furthermore, once @file{my-edb-setup.el} provides (or arranges to provide)
feature @code{my-edb-setup}, it is no longer loaded on subsequent connections.
Contrast this with the second and third way, which redefine
@code{my-write-record} on every connection.  The first three ways allow a
breakpoint to be set (using @code{edebug}, for example) on the function.
The fourth way avoids burdening the already-crowded Emacs namespace by making
use of an anonymous function.

@item Saves the control properties (how and where presently not described
in this manual, but maybe in the future...).
@end enumerate


@node Kinds of control property values
@section Kinds of control property values
@cindex kinds, control property values
@cindex characteristics, control property values

This section describes some characteristics common to the values of certain
control properties.  (We resist using the term ``type'' here because that term
is extremely overloaded in EDB, Emacs, and indeed throughout all of computer
science and programming.)

@macro cpropvaltype{name}
@item
@cindex \name\ (type)
@end macro

@itemize
@cpropvaltype function whose arglist specifies field names
Some properties (for instance, @code{:write-record}) specify a function
whose @dfn{arglist specifies field names}.  That is, the values of the
specified fields of the current record are bound to function parameters with
the same name.

@cpropvaltype text-block specification
Some properties (for instance, @code{:display}) specify a @dfn{text-block
specification}, which is either a plist that specifies how to further process
inline data, or @code{t} which expands to the default spec.  Properties of the
plist are: @code{:EOTB}, a string (default ``:EOTB'') that delimits the text
block when appearing on a line by itself; @code{:name}, a string that names
this text block, or @code{t} (default); @code{:coding}, a symbol that
specifies the coding to use when reading the text block, or @code{t} (default)
which means use the current buffer's coding.

[EXPERIMENTAL / DO NOT RELY ON THIS: Additionally, for control property
@code{:data}, the text-block spec must include @code{:seqr}, a symbol naming a
function defined in the @code{single} schema-schema.]

Here is an example showing two @code{:display} specifications, both named, and
the latter using @samp{foo} as the end-of-text-block indicator:

@example
:display (:name "time first")
At \time\ , we need to be at \place\ .
:EOTB

:display (:name "place first" :EOTB "foo")
Let's try to be at \place at \time\ , for once!
foo
@end example

@cpropvaltype string or regexp vector
Some properties specify a @dfn{string or regexp vector}.  This means the value
is either a string, or a vector of either two or three elements, whose first
element is a regular expression (string) and whose second element is an
integer specifying the relevant submatch number (use 0 to mean that the text
matched by the entire regexp is relevant).

If the vector has three elements, the third element is a string used when
writing.  To maintain format integrity (to be able to read back what is
written), this string should match the regexp specified in the first element.
In other words:

@lisp
(let ((v [REGEXP SUBMATCH WRITE-STRING]))
  (string-match (aref v 0)
                (aref v 2)))
@end lisp

should have a non-nil value.  Be careful!  (At this time, EDB does not check
this relationship; you have to check it yourself.  Omitting or specifying the
wrong @var{write-string} silently corrupts the data during write.  This laxity
ostensibly supports read-only interaction...)

@cpropvaltype record initializer function
Some properties specify a function that must @dfn{return a record initializer}.
This means the return value is either a plist of alternating field names
(symbols) and values, or a list whose @sc{car} is the keyword @code{:alist},
followed by an alist associating field names and their values.
@end itemize

@unmacro cpropvaltype


@node Control properties reference
@section Control properties reference
@cindex control properties reference

This section describes all the control properties recognized in the
@code{single} schema-schema, grouped according to typical usage.
@xref{Kinds of control property values}, for information on common
characteristics.

@macro cprop{name,rest}
@item \name\ \rest\
@vindex \name\ control property
@end macro

@subsection Minimal required

Most control properties are optional; however, a minimal control requires
@code{:display} and either @code{:fields} or @code{:tagged-setup}.

@table @code
@cprop{:display, TEXT-BLOCK-SPEC}
This control property is required.  @xref{Display specifications}.

There may be more than one @code{:display} control property, in which case
the respective @var{text-block-spec} should include @code{:name STRING}, and
you should specify @code{:choose-display} to select one of the formats.

@cprop{:fields, VECTOR}
Each @var{vector} element is either a field name or a cons whose @sc{car} is a
field name and whose @sc{cdr} is a type name (both symbols).  Either
@code{:fields} or @code{:tagged-setup} must be specified, but not both.
@xref{Record field types}.

@cprop{:tagged-setup, PLIST}
@var{plist} must include @code{:fields (TAGGEDFIELDSPEC ...)}, where each
@var{taggedfieldspec} has the form @code{(FIELDSPEC TAG DESCRIPTION)}.
@var{fieldspec} can be either a field name, or a cons whose @sc{car} is a
field name and whose @sc{cdr} is a type name.  @var{tag} and @var{description}
are both strings.  @xref{Tagged file layout}.  Either @code{:fields} or
@code{:tagged-setup} must be specified, but not both.
@xref{Record field types}.
@end table

@subsection File layout

Several control properties pertain to I/O, i.e., reading and writing to an
external data source/sink.  Note that some of these override others.
@xref{Database file layout}.

@table @code
@cprop{:read-record, FUNCTION}
@var{function} is repeatedly called with no args to parse the accessible
(narrowed) region of the data file, to return a record initializer for the
data represented in that region.  When called, point is at each region's
point-min.  If specified, control property @code{:field-separator} is ignored,
since @var{function} does all the parsing work.

@cprop{:write-record, FUNCTION}
@var{function}, whose arglist specifies field names, is called to insert into
the current buffer at point the external representation of the current
record's data.  The arglist may also be nil, in which case all field values
are dynamically bound to variables with the same names as the fields.

@cprop{:record-separator-function, FUNCTION}
@var{function} takes a buffer position, the end of the previous
separator (that is, the start of the current record or field),
and returns a pair of two buffer positions bracketing the next
separator.  That is, the returned values are the end of the current
record or field and the beginning of the next one (or nil if there are
no more).  When @var{function} is called, point is at the beginning of an
item and the buffer is narrowed to the region being currently processed.
If specified, @code{:record-separator} and @code{:record-terminator} are
ignored.

@cprop{:record-terminator, STRING-OR-REGEXP-VECTOR}
If specified, takes the place of (overrides) @code{:record-separator} and
additionally specifies that @var{string-or-regexp-vector} appears after the
last record, as well as between records.

@cprop{:record-separator, STRING-OR-REGEXP-VECTOR}
The default is the string consisting of one newline (ASCII 0xA) character.

@cprop{:field-separator, STRING-OR-REGEXP-VECTOR}
The default is the string consisting of one tab (ASCII 0x9) character.

@cprop{:substitution-separators, [FSEP RSEP]}
A special case of @code{:substitutions} (below), @var{fsep} and @var{rsep}
(both either a string or nil) specify what to write to disk in place of the
regular field and record separators, respectively, in the case where those
separators are found in the data itself.

If unspecified, values are automatically chosen.  (How?)

@cprop{:substitutions, VECTOR}
Each element in @var{vector} is a cons of two strings, specifying
substitutions on the data to be made on read (beginning to end, s/car/cdr/)
and write (end to beginning, s/cdr/car/) operations.  Note: use
@code{:substitution-separators} to disambiguate field and record separators
instead of this control property.

@cprop{:cruft, [[BEFREC AFTREC] [BEFFLD AFTFLD]]}
This is a catch-all for specifying irregular file formats.  The value is a
vector of two elements, each in turn a vector of two elements.  Each of
@var{befrec} (before record), @var{aftrec} (after record), @var{beffld}
(before field) and @var{aftfld} (after field) is a string or regexp vector.

@cprop{:data, TEXT-BLOCK-SPEC}
This is an EXPERIMENTAL control property that specifies that the data is
inherent in the control (specifically, in the text block following this
control property) instead of residing elsewhere.  To ``save'' inherent
data, EDB updates the text block and, if the control is associated with
a file on disk, saves the control to disk.

A control with inherent data can be useful for read-only and/or
dynamically generated data sets.  At present, inherent data usage is
extremely stylized:

@itemize
@item @var{text-block-spec} must include @code{:seqr read-line},
and the text block must consist of one vector, representing a record,
per line.  EDB does not validate the record for well-formedness.

@item @var{text-block-spec} should include @code{:seqw write-line},
if the data is to be saveable.  (You may omit this if you plan to use
the data in a read-only manner.)

@item If the control is not visiting a file, it must not be deleted
for the lifetime of the data access.  If the control is a file-visiting
buffer, on the other hand, you can delete it without harm, once EDB is
done processing it and has set up a data display buffer.

In either case, you should take care to not modify the control while its
data is under EDB's care, for that would probably confuse EDB when it
tries to update the text block.  Result: not only data corruption (bad),
but control corruption (very bad), as well.
@end itemize
@end table

@subsection Display

Some control properties affect what you see in an Emacs buffer.
@xref{How information is displayed}, and @code{:display} (above).

@table @code
@cprop{:choose-display, FUNCTION}
@var{function}, whose arglist specifies field names, is called immediately
prior to display a record in the data display buffer.  It should return a
string naming one of the display formats defined with @code{:display}.  This
function overrides that of @code{:before-display}.

@cprop{:before-display, FUNCTION}
@var{function}, whose arglist specifies field names, is called immediately
prior to displaying a record in the data display buffer.  This control
property is ignored if @code{:choose-display} (above) is specified.

@cprop{:report, TEXT-BLOCK-SPEC}
This specifies a report template.  @xref{Reports}.

@cprop{:summary-format, STRING}
@var{string} is a concise (preferably one line) template for the summary
buffer.  @xref{Database Summary mode}.
@end table

@subsection Miscellaneous

@table @code
@cprop{:name, STRING}
@var{string} is used for informational purposes (messages and prompts).
If unspecified, EDB assigns the default ``Unnamed Database N'', where @var{n}
is some positive integer.

@cprop{:first-change-function, FUNCTION}
@var{function} is called the first time a field changes value (due to editing)
with three args: the field name (a symbol), the old value, and the new value.

@cprop{:every-change-function, FUNCTION}
@var{function} is called every time a field changes value (due to editing)
with three args: the field name (a symbol), the old value, and the new value.

@cprop{:field-order, VECTOR}
@var{vector} specifies the priority of significant fields for a sort operation
(elements towards the beginning have precedence over those towards the end).
Each element is either a field name or a cons whose @sc{car} is a field name
and whose @sc{cdr} is a symbol: @code{ascending} or @code{descending}, which
specifies that the field is to be sorted in ascending or descending order,
respectively; or a cons
of @var{type} and @var{value}, where @var{type} is a symbol (either
@code{order-function} or @code{sort-function}) and @var{value} specifies
the function.

If unspecified, the default field order is taken to be the same as
in @code{:fields}, i.e., all fields significant and sorted in
ascending order.  @xref{Sorting}.

@cprop{:locals, VECTOR}
Each element of @var{vector} is either a symbol, or a list of the form
@code{(@var{symbol} @var{init-form})} --- like an entry in a
@code{let}-bindings block.  These define variables (mostly opaque to EDB)
unique to the connection.  The associated value is the value of
evaluating @var{init-form} (if the element is a cons), otherwise nil.

@cprop{:record-defaults, FUNCTION}
@var{function} is called with no args when first adding a record, to return a
record initializer.
@end table

@unmacro cprop

@node New control example
@section New control example
@cindex new control example
@cindex example, new control

This section describes a simple example database and the steps required to
construct an appropriate control buffer to connect to it.

First, we must decide how many fields the database shall contain, and
their names and types.  The names are Lisp symbols, and cannot be
either @code{t} or @code{nil} for implementation reasons (those symbols
cannot be used as variable names in Emacs Lisp).
Let us make a name database with three string fields called
@samp{first}, @samp{middle}, and @samp{last} and one integer field
called @samp{age}.

Next, we must decide how a record is to be displayed in an Emacs buffer.
Let us try the following:

@example
Family: \last
Given:  \first
Middle: \middle

Age:    \age
@end example

The backslash followed by a field name indicates that the field's contents
should be inserted there.  For more information about specifying display
formats, @xref{How information is displayed}.

Lastly, we need to compose a control buffer comprising a standard header,
and the two pieces of information we've decided upon.  So we type:

@itemize
@item @kbd{C-x b my-names.edb @key{RET}}
@item @kbd{M-x emacs-lisp-mode @key{RET}}
@end itemize

@noindent
and insert the following (without the indentation):

@example
:EDB (single) ;;; -*- emacs-lisp -*-

:fields [first middle last (age . integer)]

:display t
Family: \last
Given:  \first
Middle: \middle

Age:    \age
@end example

(Alternatively, we can insert the file @file{names.edb} from the EDB
distribution's @file{examples} subdirectory, or simply visit it.)

Now we can type @code{M-x edb-interact @key{RET}} in this buffer,
typing @key{RET} to specify this buffer as the control, and
@file{my-names.data} (or any other non-existent filename) to specify the data.
An ``empty'' record will be shown, ready to be filled in.
@xref{Database mode}.

(Alternatively, we can specify the file @file{names.dat} from the EDB
distribution's @file{examples} subdirectory, in which case the record for
Harry S. Truman (age 88) will be shown.)

When we write out the data, fields and records will be separated by
tab and newline, respectively.  These are the default delimiters.

For more information about customizing EDB for your application,
@pxref{Customization}.

Once we are happy with the control buffer, we can save it in a file for later
use; we can then specify that file as the control for a subsequent
@code{edb-interact} session.


@node Ongoing migration
@section Ongoing migration
@cindex ongoing migration
@cindex control without control properties

Although most control can be declaratively specified, there are still some
noticible holes (@pxref{Control properties reference}), which we will endeavor
to close as we proceed towards EDB 2.x.

Notably, to extend the recognized types requires calling functions.
@xref{Specifying a record field type}, for record field types.
@xref{Defining new displaytypes}, for display types.

Also, several variables (that are not set automatically when some control
property is specified) are available for tweaking with @code{setq} and/or
@code{add-hook}:

@table @code
@item db-before-read-hooks
@itemx db-after-read-hooks
@xref{Read hooks}.
@end table


@node Changing control properties at runtime
@section Changing control properties at runtime

You can use @code{edb-get} and @code{edb-put} to access and modify,
respectively, the value of certain control properties at runtime
(and only at runtime, presently).

@defun edb-get property &optional buffer
Return the value associated with @var{property} for the current buffer.
Optional second arg @var{buffer} specifies another one to use as ``current''.
@end defun

@defun edb-put property newvalue &optional buffer
Set @var{property} to have @var{newvalue} for the current buffer.
Optional third arg @var{buffer} specifies another one to use as ``current''.
@end defun

The following table describes the available control properties.  The
default value is the one used when setting up the buffer.

@table @asis
@item @code{:wraparound} (default: @code{delay})
This controls motion between the first and last record (and vice versa),
when moving sequentially between records.  A value of @code{nil} means
do not wrap around; stop at the first or last record and display a
message to that effect.  A value of @code{delay} (a symbol) means to
stop at the border, displaying a message, but wrap on the next attempt
to move, anyway.  A value of @code{t} means to wrap without stopping,
and without displaying a message.

@item @code{:stay-in-edit-mode-p} (default: @code{t})
A non-@code{nil} value means that when editing a record, moving to the
next record stays in Edit mode.
@end table


@node Example control file from scratch
@section Example control file from scratch

@cindex WWW Links control file examples
@cindex control file examples
This section describes three example databases of increasing complexity and
their associated control files.  The data is a set of ``WWW Links''.  We start
with just the URL and add fields/features as we go along.  All files are
available in the distribution's @file{examples} subdirectory.

@subsection WWW Links

@dfn{WWW Link} is a fancy way to say ``URL''.  The simplest database is just a
list of these in a file, one per line.  For example:

@example
http://www.gnu.org/
http://www.gnu.org/software/
http://www.gnu.org/software/emacs/
http://www.gnuvola.org/
http://www.gnuvola.org/software/
http://www.gnuvola.org/software/edb/
@end example

It's easy to construct a minimal @file{www-links.edb} to grok this (note that
(parenthesized) line numbers are for discussion, and should not appear in the
actual working file):

@example
:EDB (single)           (1)
:name "WWW Links"       (2)
:fields [url]           (3)
:display t              (4)
  \url                  (5)
:EOTB                   (6)
@end example

Line 1 is boilerplate and will be assumed henceforth.  Line 2 names the
database.  Line 3 declares that each record is composed of one field, whose
name is @code{url}.  The field type's is taken (implicitly) as @code{string}.
Lines 4 through 6 describe the per-record display.  The @code{\url} in that
text block means to display the field @code{url} at that position.

That's it!  Use @kbd{M-x edb-interact RET examples/www-links.edb RET RET} to
try it out.  (The last @kbd{RET} is to let EDB find the data file, which is
@file{examples/www-links} by default.  You can, of course, specify another
data file to use instead of the default.)

@subsection WWW Links (Titled)

In this subsection we extend the simple database introduced in the previous
subsection to add a @code{title} field.  Since the line-oriented (one record
per line) approach is pleasant enough, the only design decision we have to
make is how to separate the fields in each record, and their order.

Although @code{TAB} (aka @code{^I} aka @code{C-i} aka ``control i'') is the
traditional (and thus default) field separator, that might cause problems if
that character were to appear in the title.  We choose @code{^T} instead,
since it is virtually guaranteed not to conflict, and ``T'' is a workable
mnemonic for ``title'' (bonus!).

As for the order of fields in the data file, let's keep the URL first
since it is most important.  (Although a title without a URL might be useful
as a placeholder, our goal is to organize ``links'', which implies a certain
connectedness requirement.)

To effect this new design in a new data file, we can evaluate the form:

@example
(with-temp-file "www-links2"
  (insert-file-contents "www-links")
  (while (< (point) (point-max))
    (end-of-line)
    (insert "\C-t")
    (forward-char 1)))
@end example

Next we copy @file{www-links.edb} to @file{www-links2.edb} and make the
following modifications:

@example
:name "WWW Links (Titled)"      (1)
:fields [url title]             (2)
:field-separator "\C-t"         (3)
:display t                      (4)
  \title                        (5)
  \url                          (6)
:EOTB                           (7)
@end example

Line 1 updates the title.  Line 2 declares the ordering in the data file and
names the second field @code{title}, with type (implicitly) @code{string} as
for @code{url}.  Line 3 declares the field separator.  Lines 4 through 7
describe the per-record display.  Note that in the display we place the title
first and on a line by itself.  This has implications in the summary
(@pxref{Database Summary mode}), which defaults to the first line in the
@code{:display} text block with a displayspec in it.

With these changes, we can @code{edb-interact} with @file{www-links2.edb} and:
type @kbd{e} once to enter edit mode; type in a title; use @kbd{M-n} and
@kbd{M-p} to move to next/previous records, staying in the @code{title} field;
type @kbd{C-x C-s} to save and @kbd{x} to exit.

@subsection WWW Links (Everything)

In this subsection, we further extend the database to include two fields:

@itemize
@item date when the URL was last checked for reachability;
@item comments (notes, critiques, etc)
@end itemize

For the date, we use a custom data type @code{suredate}, defined as:

@example
(edb-define-recordfieldtype 'suredate 'date
  :actual->stored 'edb-t-timedate1:format-date-iso
  :stored->actual 'edb-t-timedate1:parse-date-string)
@end example

Comments are just another string, but there is a catch: we would like to
support multi-line comments, such as:

@example
This site is ok, but I wouldn't want to have
to maintain it, for these reasons:
 - probably doesn't use EDB for its database
 - requires javascript, which w3m doesn't do
@end example

The implication is that we can no longer use one record per line, as some
records may contain the newline character in the @code{comments} field.  So,
let's choose formfeed (aka @code{^L} aka @code{C-l} aka ``control l'').

To effect this new design in a new data file, we can evaluate the form:

@example
(with-temp-file "www-links3"
  (insert-file-contents "www-links2")
  (while (< (point) (point-max))
    (end-of-line)
    (insert "\C-t\C-t\C-l")
    (if (= (point) (point-max))
        (delete-char -1)
      (forward-char 1))))
@end example

Next, we create @code{www-links3.edb}, modifying the control property
@code{:name}, as usual, and adding @code{edb-define-recordfieldtype} as
described above.  The rest of the file looks like:

@example
:fields [url title (last-check . suredate) comments]
:record-separator "\C-l"
:field-separator "\C-t"

:display t
  \title
  \url
  (\last-check,date-iso)

Comments:
\comments
:EOTB

:summary-format "\\last-check,width=11 \\title"
@end example

Note that the @code{last-check} field is declared with type @code{suredate} in
@code{:fields}, is displayed with the @code{date-iso} format in the
@code{:display} text block; and is explicitly included (with maximum width of
11 columns) in @code{:summary-format} (the double backslashes are required
because the value is an Emacs Lisp string).


@node Record field types
@chapter Record field types
@cindex record field type
@cindex field type, record
@cindex type, record field

All the records of a database contain fields of the same types: the
fifth field might always contain an invoice number, for instance, but
the invoice number would vary from record to record.  This chapter
describes how to specify and use different record field types.

The most important information about a record field is

@itemize @bullet
@item display representation onscreen and in reports
@item EDB's internal representation
@item storage representation in data files
@item how to convert among these representations
@item how to sort items of that type
@end itemize

@cindex record field type, compared to displaytype
@cindex displaytype, compared to record field type

This information is separated into a display specification and a record
field specification.  The display specification determines how a field's
contents are displayed and parsed onscreen (say, in a data display
buffer).  The record field specification controls everything else about
the record field; its information does not depend on the onscreen (or
in-report) visual appearance of the field.  The control
specifies a displaytype for each display specification in the format
file (that is, for each location in the data display buffer that will
contain a representation of some record field), and
specifies a record field type for each field in a database record,
whether or not the field is ever displayed.  (A particular field's
contents may appear zero, one, or more times in a data display buffer;
one displayspec structure is created for each occurrence.)

Displaytypes and record field types are distinct; they supply
complementary information.  There is not even a one-to-one relation
between them.  A particular record field type may be displayed in any of
a number of ways by using different displaytypes --- dates are such an
example.  On the other hand, record field types which are interpreted,
sorted, and stored on disk differently, but which have the same internal
representation --- say, as a string --- could all be displayed and edited
using the same displaytype.

This chapter discusses record field types, record field specifications,
and lists the attributes upon which the other two are built.
For more information
about displaytypes, display specifications, and the displayspec
structure, @pxref{How information is displayed}.

@menu
* Specifying a record field type::
* Predefined record field types::
* Record field attributes::
@end menu


@node Specifying a record field type
@section Specifying a record field type
@cindex specifying a record field type
@cindex record field type, specifying
@cindex type, specifying record field

The function @code{database-set-fieldnames-to-list}
specifies the types (and names) of record fields.  This function
is called automatically when either of the control properties @code{:fields}
or @code{:tagged-setup} is specified.

@defun database-set-fieldnames-to-list database fspecs [dtype]
Set @var{database}'s field names and record field types according to
@var{fspecs}, canonicalizing and caching internally various things,
but do nothing if the information has already been canonicalized.

@var{fspecs} is a list, each element of which can either be a single symbol,
the field name, or a cons @code{(NAME . RECORDFIELDTPYE)}.  Optional third arg
@var{dtype} specifies the default recordfieldtype for single-symbol elements.
If @var{dtype} is not given, use the value of @code{db-default-field-type}
if bound, or @code{string} otherwise.
@end defun

@vindex db-default-field-type
Some notes on the deprecated variable @code{db-default-field-type}: There has
been some confusion over usage of this variable; the ideas of ``default
value'' and of ``buffer-local variable'' do not mix well together, especially
when the value in this case is the recordfieldtype of the field, a property of
the field and not of its display mechanism or of any buffer.  Thus, if you
have old code that sets this variable, you should change it to pass that
value as the third argument to @code{database-set-fieldnames-to-list}.

A record field type gives information about one field of the database's
records: the type of the contents, what sorting function to use, how to
write it to disk and read it back, constraints on its value, etc.
Each record field has a record field type.

Usually one of the predefined types suffices, but not always.  You can use
@code{edb-define-recordfieldtype} to create and install a new
record field type.  (The @code{-from-recordfieldspec} in the function name is
a historical artifact; before being extended, the function used to only accept
a recordfieldspec as @var{source}.)

@defun edb-define-recordfieldtype name source [override...]
Define a recordfieldtype @var{name} (a symbol) with @var{source}.
@var{source} may be a recordfieldspec, the name (a symbol) of a currently
defined type, or @code{nil}.  In all cases, a new recordfieldspec object
is created and then modified by @var{override}, a sequence of alternating
keywords and values, and finally added to a global list.  Return @var{name}.
@end defun

@xref{Predefined record field types}, for a list of symbols useful for
the @var{source} argument.  @xref{Record field attributes}, for a list
of keywords useful as part of the @var{override} argument.

After specifying the field names and types, it is usually a good idea
to specify some field-specific help, as well.  This kind of help is kept
separate from the @code{:help-info} of each recordfieldtype
(@pxref{Record field attributes}), since several different fields
(requiring different help info) may specify the same recordfieldtype.

@defun db-set-field-help database field1 help-info1 [...]
Set field-specific help info for @var{database} from @var{field1}
@var{help-info1}, @dots{}.  @var{field1}, @var{field2}, @dots{}
should be field names (symbols) or field numbers, while
@var{help-info1}, @var{help-info2}, @dots{} should be either a string or
a form to be evaluated that results in a string.
@xref{Getting help}.

This should be called after the field names have been set up.
@end defun

Examples of record field type creation and installation can be found in the
file @file{db-types.el}, the source for most of the predefined types.

Record field types should not be confused with display types; a display
type is used to specify how a particular value is shown on the screen,
but a record field type constrains the information actually contained in
the record field.


@node Predefined record field types
@section Predefined record field types
@cindex record field types, predefined
@cindex predefined record field types

The following record field types are predefined for your convenience; you can
redefine them (they are not ``privileged'' in any way) or even add new record
field types (@pxref{Specifying a record field type}).

If you find any of these typenames cumbersome, you can create your own
aliases for them using @code{edb-define-displaytype} or
@code{edb-define-recordfieldtype}.

@macro rftype{name}
@item \name\
@cindex \name\ record field type
@end macro

@subsection Builtin

These record field types are available as soon as EDB is loaded.

@table @code
@rftype integer
Ordinary integers.

@rftype integer-or-nil
Integers or @code{nil}, the empty value; by default, @code{nil} is
treated as larger than any integer, so it comes last in an
increasing-order sort.

@rftype number
@cindex floating-point number record field type
Ordinary numbers.  A number is either an integer or a floating-point number.

@rftype number-or-nil
Numbers or @code{nil}, the empty value; by default, @code{nil} is
treated as larger than any number, so it comes last in an
increasing-order sort.

@rftype boolean
This displayspec corresponds to the yes-no displaytype.  For the
purposes of sorting, @code{t} is considered less than @code{nil}, so it
appears first in an increasing-order sort.

@rftype string
Ordinary strings.

@rftype one-line-string
Strings which may not contain newlines.

@rftype string-or-nil
Either a string or the value @code{nil}, which is converted to the empty
string.  Sorting treats @code{nil} identically to the empty string.

@rftype nil-or-string
Identical to the string-or-nil record field type (except for the name).
This exists so that display fields of type nil-or-string can
conveniently default to this record field type.

@rftype one-line-string-or-nil
The obvious combination of the one-line-string and string-or-nil
record field types.
@end table

@subsection From @file{edb-t-timedate1.el}

Loading @file{edb-t-timedate1.el} provides some record field types
related to the times and dates.  The ``1'' is for upward-compatibility
with a future time/date library variation that will make use of Emacs'
intrinsic time/date support.  (@file{edb-t-timedate1.el} was written a
while back, before such support was added to Emacs, and may become
unavailable once again in the future.  See its Commentary section
for more info.)

@table @code
@rftype date
A date which specifies zero or more of the year, month, and day.  By
default the date is sorted by year, then month, then day; an unspecified
component is larger than any specified component.  For example,
@code{March 14, 1967} would
appear before @code{January 1} if dates were sorted in increasing order.
Dates are read from database files using a function that
can parse nearly any string representation of a date
and return a date object if it is passed one.  Dates
are written using a function which produces a string of
the form @code{March 14, 1967}.

@rftype date-or-nil
A date, or nil.

@rftype date-efficient-storage
@findex edb-t-timedate1:storage-string->date
@findex edb-t-timedate1:simple-format-date
When the dates in a database file are known to have a particular format,
using @code{edb-t-timedate1:parse-date-string} is unnecessarily inefficient.
The date-efficient-storage record field type specifies that
@code{edb-t-timedate1:storage-string->date},
which can efficiently read dates written by
@code{edb-t-timedate1:simple-format-date}, be used instead.
The time savings is
noticeable on large databases.

@rftype time
A time.
@end table

@unmacro rftype

@node Record field attributes
@section Record field attributes
@cindex record field attributes
@cindex fieldspec, see record field attributes

Record field attributes pertain to the content of a record field,
but do not address how it is displayed on the screen or read from
interactive input.  For that, @xref{How information is displayed}.

Here is a table of valid attributes:

@macro rfattr{name}
@item \name\
@cindex @code{\name\} record field attribute
@end macro

@table @code
@rfattr :type
A symbol such as @code{string} or @code{integer}, the type of the data
described by this collection of attributes.

When no displaytype is explicitly specified in a display specification,
then a displaytype with the same name as @code{:type} is used by
default; this is the only use for this attribute.

@rfattr :default-value
The default value for fields described by this collection of
attributes; used when creating new records.

@rfattr :common-form-function
A function which, called on the contents of a record field, returns them
in canonical form.  This can be used for determining non-trivial
equality, when two nonidentical values should be considered equivalent.

@rfattr :merge-function
A function which, called on the contents of two record fields, returns a
combination of the two.  It may be interactive.

@item :order-function
@itemx :sort-function
@cindex @code{:order-function} record field attribute
@cindex @code{:sort-function} record field attribute
@findex database-sort
The record field's ordering and sorting functions (@pxref{Sorting
functions}).  Both ordering and sorting are possible if either
attribute is specified.

If both attributes are unspecified,
then a dummy ordering or sorting function is
used, so sorting on this field has no effect.  Since the function is
called and its result examined, this is more expensive than not sorting
on the field in the first place.  If it does not make sense to sort on a
particular field, then it is best to keep that field out of the field
priorities used for sorting (which is the @code{:field-order} control
property, or is interactively specified through the database sort
interface (@pxref{Sorting}), or is specified as an argument to
@code{database-sort}).

Actually, the record field type does not include order-function or
sort-function information directly; they are built upon the value
of the @code{:field-order} control property.

@rfattr :match-function
A function which takes a pattern and a field value and returns
non-@code{nil} if they match.  The function should also be able to take
as its first argument a field value rather than a pattern.

@rfattr :help-info
A string which is displayed by @code{db-field-help} when there is no
field-specific help available.  Field-specific help is usually
preferable to this help, which only describes the type of the field's
contents.

@rfattr :actual->stored
A function which converts a field value into its on-disk representation
(a string).

@rfattr :stored->actual
A function which recovers a field value from its on-disk representation
(a string).  If this function returns a string, it should return
something reasonable if supplied the empty string as its argument.
(That can happen when an empty database is read.)

@rfattr :constraint-function
@cindex enforcing constraints
A function which the value of this field must satisfy; that is, the
function must return non-@code{nil} on it.  The function may reject the
value either by returning @code{nil} or by signalling an error; the
latter permits the function to provide an informative message about
the problem.

Four arguments are supplied to @code{:constraint-function}: the field
value, the record, the record fieldnumber, and the database.  This
permits cross-field and cross-record constraints.  The record argument
may be @code{nil}, in which case the function should return @code{t} if
the value is acceptable for some conceivable record.  This occurs, for
instance, when values are read in a call to
@code{db-field-query-replace}.

The constraint function may be interactive; for instance, it may
give you the opportunity to override the constraint.

@end table

@unmacro rfattr


@node Database file layout
@chapter Database file layout
@cindex file layout for data file
@cindex file format for data file
@cindex database file layout
@cindex data file layout
@cindex file layout
@cindex layout, of data file
@cindex format, of data file
@c @cindex writing to disk
@c @cindex saving to disk

This chapter discusses specifying how a database is read from a file (or
saved back to it).

Broadly stated, there are three possible file representations for a
database: EDB's internal file layout, a delimited layout, or a nonregular
layout.  EDB's internal file layout is designed for fast reading and
writing, but is not very human-readable.  A delimited layout is one in
which records (and fields within a record) are separated from one
another in a predictable (though not necessarily invariant) way.  A
nonregular layout is any other kind of layout; you may specify arbitrary
Emacs Lisp code to read and write such files.  EDB also supports tagged file
layout (a special case of nonregular file layout).

If the database is to be stored in EDB internal file layout, a lot of
this information is not needed except when the database is first
created.

The sections of this chapter each describe a file layout, except the
last, which describes in detail the process of reading a database from
disk.


@menu
* Data encoding::
* Internal file layout::
* Delimited file layout::
* Tagged file layout::
* Relational file layout::
* Nonregular file layout::
* Reading from disk::
@end menu


@node Data encoding
@section Data encoding
@cindex data encoding
@cindex encoding, data
@cindex coding system

When connecting with @code{db-find-file}, the format file can specify that
coding system to be used for the data by setting @code{edb-data-coding}.  This
variable is not meaningful when control is specifed using control properties.

@defvar edb-data-coding
A symbol in the Local Variables section of the format file specifying
the coding system Emacs should use when reading and writing the data
file.  This coding system may be different from the one used for the
format file.
@end defvar


@node Internal file layout
@section Internal file layout
@cindex file layout, internal
@cindex data file layout, internal
@cindex internal data file layout

EDB 1.x supports an @dfn{internal file layout}, but EDB 2.x will not.  This
section details the layout, as well as how to migrate away from it (should you
plan to move to EDB 2.x at some point in the future).

@menu
* Internal file layout details::
* edb-1int-to-single::
@end menu


@node Internal file layout details
@subsection Internal file layout details

The first line of a database file in EDB's internal file layout looks
something like

@lisp
;; Database file written by EDB; format 0.7
@end lisp

followed by two Emacs Lisp forms: a vector (the database structure),
and an alist of various miscellaneous attributes.
The rest of the file is a sequence of vectors (the records of
the database).

Databases stored in this layout can be read and written
very quickly (sometimes orders of magnitude faster than databases which
EDB must parse when reading), and they never suffer from ambiguities
between data and delimiting text, but they are not easy for people to
read and understand.  A human- or program-readable version of the
database can be generated when it is needed, either by creating a report
or by saving in some other file layout.  This is a good option when all
manipulation of a database will be done via EDB.

@cindex EDB internal layout, converting to or from
@cindex converting a file to or from EDB internal layout
Since this file layout is rather complicated, databases are often
created in some other file layout and then converted to this one.  To
convert from another file layout to EDB's internal file layout, read in
the database, use @code{db-toggle-internal-file-layout}
to mark it for saving in that format,
and then write or save the database (via @kbd{C-x C-w} or
@kbd{C-x C-s}).

@deffn {Command} db-toggle-internal-file-layout
Toggle whether the database will be saved in EDB's internal file layout.
With a nonzero prefix argument, set it to use internal file layout.
With a zero prefix argument, set it not to use internal file layout.
@end deffn

After a database has been saved in internal file layout, then any forms
in the auxiliary or format files that set database slots can be removed
if desired; this is not necessary, however.

Converting a database from EDB file layout to some
other representation is similar, and certain variables and
database values may need to be set (see the documentation for the layout
you desire, elsewhere in this chapter).  Making a report
(@pxref{Reports}) can also produce a different file layout for a
database, with even more flexibility than the techniques described here.


@node edb-1int-to-single
@subsection edb-1int-to-single

EDB 2.x will not support internal file layout.  Instead, it will have (and EDB
1.26 and later already have some) support for control with @dfn{inherent data}
via the @code{:data} control property (@pxref{Control properties reference}).
To ease migration, you can use @code{edb-1int-to-single} to do most of the
tedious translation work, although you may need to help it a little.

@macro FF
@code{<FIXME>}
@end macro

@deffn Command edb-1int-to-single filename
This command translates the contents of @var{filename} to a ``single''
schema-schema.  If the contents are not in EDB 1.x ``internal file layout,
format 0.7'', it signals an error.  Otherwise, it leaves the result of the
translation in a newly created output buffer named @kbd{:EDB (single) from
FILENAME}.

To compute the control properties written in the output buffer,
@code{edb-1int-to-single} uses various (nefarious) methods to parse and
repurpose the contents of two files: the one named by @var{filename}, as well
well as the primary format file (if it can be determined --- see below).
Additional format files and the auxiliary file, if any, are @emph{not}
consulted.
@end deffn

The local variables block in the primary format file can have either
@code{eval: FORM} pseudo-variables or @code{VAR: VAL} specifications.
If an @code{:eval} form looks like a function call for one of these functions:

@lisp
database-set-fieldnames-to-list --- "redundant" comment
dbf-set-summary-format --- translate to :summary-format
@end lisp

then @code{edb-1int-to-single} handles it specially (as indicated).  An
informative comment is also output to help keep track of things.  If, on the
other hand, the @code{eval:} form is not recognized, it is added to the output
anyway, but commented out.

Similarly, these variables (@code{VAR: VAL} in the local variables block) are
handled specially:

@lisp
db-new-record-function
dbf-first-change-function
dbf-every-change-function
dbf-before-display-record-function
dbf-format-name-spec-alist
edb-data-coding
@end lisp

(i.e., commented out), and all others translated into @code{:locals}.

Here are all the control properties written.  Those in the first column are
always written, while those in the second are conditionally written.

@example
:name             :field-priorities
:fields           :summary-format
:display          :substitution-separators
:data             :substitutions
                  :locals
@end example

These short lists show that there are many control properties that
@code{edb-1int-to-single} does not even attempt to infer.  Indeed, even when
it tries its best, the output buffer is likely to contain several @FF tokens,
indicating places where further attention (a nice way to say ``manual
tweaking'') is required to complete the translation.

Following is a complete table of possible @FF occurances, with the reason for
the @FF and the suggested remedies.  Note that suggestions involving editing
the output buffer by hand are not compatible with those involving iteratively
tweaking something in the input file or environment and re-invoking
@code{edb-1int-to-single}, since re-invocation clobbers hand-edits.  Thus, a
good strategy would be to do the upstream tweaking fixes first, saving the
hand-editing fixes for last.

@table @asis
@item @code{coding:<FIXME>} in the first line
This occurs when the variable @code{coding-system-for-read} is @code{nil}
during the call to @code{edb-1int-to-single}.  This is a relatively serious
situation, as recent versions of GNU Emacs may refuse to save the file
altogether since @FF does not name a valid coding system.

One way to fix this is to replace the @FF with the appropriate coding system
for the control file (i.e., resorting to editing by hand).

Another way is to call @code{universal-coding-system-argument}
(normally bound to @kbd{C-x @key{RET} c}) immediately prior to translation
(via @kbd{M-x edb-1int-to-single}).

Yet another way is to remove the @code{coding:<FIXME>;} completely, thus
foisting the responsibility of specifying the file's coding system on another
person (or mechanism) at another time.  Note that you must remove the trailing
semicolon as well.

@item @code{:name} control property
This occurs when the original name is @code{nil}, which should never happen
according to internal EDB constraints (but you never know).  To fix, choose a
suitable description for the data (a string) and replace the @FF with it.

@item @code{:display} control property
This occurs when the primary format file cannot be determined.  Normally,
@code{edb-1int-to-single} looks for the filename of readable file first in the
@code{:format-file} extra attribute, and then doing an ``EDB 1.x-style''
format-file search (@pxref{Auxiliary files}).

The easiest way to fix this is to make sure a the format file (or a symlink to
it) is named in such a way that the search can find it.  For example, if the
internal file layout filename is @file{foo.bar.baz}, make a symlink from the
actual format file to @file{foo.bar.fmt}.

@item primary format file's local variables block
As mentioned above, an unrecognized @code{eval:} form is output anyway, but
commented out.  Included in the comment is a @FF{}.  Unfortunately, no
general translation approach suggests itself since the form can be arbitrary
Lisp code.  If you are lucky, the form will be functionally redundant with one
of the written control properties and you can simply delete it.

The @code{VAR: VAL} variables listed above likewise have a @FF included in
their comment.  For @code{edb-data-coding}, you can copy its value to the
first line.  For @code{dbf-format-name-spec-alist}, you need to manually
create new a @code{:display} block for each format named [lame ---ttn].
For the others, where the control property value is a function of some sort,
you need to change the function to fit the expected calling convention of the
associated control property [explain! ---ttn].
@end table


@node Delimited file layout
@section Delimited file layout
@cindex file layout, delimited
@cindex data file layout, delimited
@cindex delimited file layout

@cindex tab-separated text file layout
EDB can conveniently read and write database files in which records are
separated from one another by a record delimiter and, within each
record, fields are separated by a field delimiter.  When the delimiters
are the newline and tab characters, respectively, the result is the
standard @dfn{tab-separated text} layout, which is often used for
transferring information from one program to another.

@cindex delimiters, record and field
@cindex record delimiters
@cindex field delimiters
The record and field delimiters need not be single characters; they can
be arbitrary strings or even regular expressions.  The latter is useful
if the exact
delimiter is not known ahead of time (for example, if records may be
separated by one or more newlines).  This regular expression
mechanism can only be used when reading the database: when writing a
database, all the record delimiters are identical, and so are all field
delimiters.  (Exception: you may specify an arbitrary record-writing
function (@pxref{Nonregular file layout}) and arbitrary functions for
either reading records or for separating records or fields in delimited
layout, but should use the simpler reading mechanisms whenever possible,
for your own sake.)

Using a delimited file layout has one important consequence: the
delimiter strings may not appear in the data, lest
those occurrences be misinterpreted as delimiters rather than as data.
For solutions to this, @xref{Resolving ambiguities}.

@menu
* How to specify delimited file layouts::
* Resolving ambiguities::
* Problems with end-of-file newlines::
@end menu


@node How to specify delimited file layouts
@subsection How to specify delimited file layouts
@cindex separator, setting record
@cindex separator, setting field
@cindex record separator, setting
@cindex field separator, setting

If the data source can be represented as text in a buffer, and is amenable to
separation using textual methods --- by recognizing a string, by matching
against a regular expression, or by calling a stylized parsing function ---
then it is said to be stored in a @dfn{delimited layout}.  Reading involves
determining record and field boundaries followed by parsing the data fields.
Writing involves inserting into an empty buffer representations of record and
field boundaries interleaved with those of the data (and then saving the
buffer to disk).

The control properties (@pxref{Control properties reference}),
in order of importance for determining record
boundaries when reading are: @code{:record-separator-function} followed by
@code{:record-terminator} followed by @code{:record-separator}.  For fields,
the only applicable control property is @code{:field-separator}.

@cindex beginning of file, text at
@cindex end of file, text at
To specify text at the beginning and/or end of the data (or at the beginning
and/or end of each record) to ignore, use the @code{:cruft} control property.

When writing, those control properties that specify a regexp vector
(@pxref{Kinds of control property values}), must
specify the last element (a literal string), since it is not possible to
reliably derive such a string from the regular expression and submatch
information given in the first two elements of the vector.

For instance, suppose a database
has records with a variable number of fields separated by newlines, that
records are also separated by newlines, and that the first field of each
record has some special form different from all other fields (say, it is
a number with a decimal part).  The following code would permit
separation of the records without writing a special function to do so
and without including the decimal number in the separating text:

@lisp
:record-separator ["\\(\n\\)[0-9]+\\.[0-9]+" 1 "\n"]
@end lisp

Be careful to use a correct value for separator strings.  For instance, if
your record separator is a form feed on a line by itself, you probably want to
set @code{:record-separator} to @samp{"\f\n"}, or possibly @samp{"\n\f\n"},
rather than just @samp{"\f"}, lest the newlines be considered to be part of
the records rather than part of the separator.

@subsubsection More examples

For instance, to parse "[Mary, John,Jack, and Jill]" and to write it
back out as "[Mary, John, Jack, Jill]", the following specification
would suffice:

@lisp
:record-separator [", *\\(and +\\)?" 0
                   ", "]
:cruft [["[" "]"]
        [nil nil]]
@end lisp

[[[ This following example should go away when writing is converted from ``all
or nothing'' to ``selective update''.

The -string slots are used for writing; but what if you only have a regexp
for the leading or trailing junk, but you want that restored exactly?
You can set pre-first-string *after* the database file has been found.
For instance, in db-before-read-hooks, use a function such as

@lisp
(defun btxdb:read-comments ()
  (save-excursion
    (set-buffer db-buffer)
    (goto-char (point-min))
    (when (search-forward "@@" nil t)
      (setf (sepinfo-pre-first-string
             (database-record-sepinfo database))
            (buffer-substring (point-min) (point))))))
@end lisp

or even put

@lisp
(setf (sepinfo-post-last-string
       (database-record-sepinfo database))
      (with-current-buffer db-buffer
        (goto-char (point-min))
        (re-search-forward "\n\C-l\n")
        (buffer-substring (match-beginning 0)
                          (point-max))))
@end lisp

as is in your auxiliary file.
]]]

If all records have five lines (say), you can use:

@lisp
:record-separator-function (db-make-n-line-sep-function 5)
@end lisp

This is also useful when both the field separator and the record
separator are the newline character.

@defun db-make-n-line-sep-function n
Return a function useful when all records have exactly @var{n} lines.
@end defun


@node Resolving ambiguities
@subsection Resolving ambiguities
@cindex resolving ambiguities in database files
@cindex ambiguities in database files, resolving

Substitution is a mechanism for dealing with the problem of
distinguishing field and record separators from the contents of database
records.  For instance, if the newline character (actually, a string
consisting of only the newline character) is used as a record separator,
and records may contain multiline text fields (or other fields whose
storage representation contains a newline), then how would EDB know,
when reading the database back in, which newlines are record separators
and which are part of fields?

There are several ways to avoid this ambiguity.

@itemize @bullet
@item
Disallow the use in record fields of the character or string causing
the ambiguity.  For instance, in the example above, you might change the
record field type of all of the string fields to one-line-string.

@item
Change the separator(s) to strings that do not appear in the storage
representation of any field.  For instance, Unix password files are
stored in delimited file layout with a colon as the field delimiter (and
colons are prohibited from appearing in the field text).

@lisp
:field-separator ":"
@end lisp

Strings containing non-printing characters are another good bet, but
this method relies on luck and the hope that the chosen separators
will never appear in data.

@item
Change the representation of the ambiguous string, when it appears in
data; this guarantees that whenever the string does appear in a database
file, it stands for a separator.  This scheme is called substitution,
because another string is substituted for the ambiguous one when it
appears in data.  This is similar to the previous workaround, which
changed the separators rather than the data-bearing instances of the
string.  Ambiguities are still possible, if the substituted text happens
to appear elsewhere in data.  Specifying a substitution is described
below.

@item
The simplest solution is to use EDB's internal file layout
(@pxref{Internal file layout}).  Ambiguities can only occur when the
field data and the separators are both text to be interpreted by EDB@.
EDB's internal file layout uses Emacs Lisp's mechanisms (a built-in form
of quoting) to ensure that what is read in is identical to what was
written out.

@end itemize

@cindex substitution, in reading a database file
Substitution is the replacement of potentially ambiguous strings by
other ones.  For instance, when writing tab-separated text, each
occurrence of the newline character in a field could be replaced by
control-k when the database is written.  Then, when the file is read in,
every newline can be safely assumed to be a record separator.  The final
step is converting the control-k characters back into newlines.
The problem with this approach is that
if there were any control-k characters in the text, then when the
database is read back in, they will be (incorrectly) converted to
newlines.  EDB warns when the database is being written out if this
problem could occur; you may choose a different substitution or abort
the database write operation.  It is usually possible to find a
substitution --- a character or sequence of characters that doesn't appear
in the data.

The @code{:substitutions} control property specifies a
vector of string pairs: @code{(DATA . FILE)}.
For example, to make control-k in the database file stand
for newline in the data, include in the control:

@lisp
:substitutions [("\n" . "\C-k")]
@end lisp


@node Problems with end-of-file newlines
@subsection Problems with end-of-file newlines
@cindex newline, at end of database file

Suppose you want to get rid of every newline at the end of the database
file, but you don't know how many there are.

@lisp
:cruft [[nil ["\n*\\'" 0 "\n"]] [nil nil]]
@end lisp

does not work, because the post-last-record regexp is searched for
backward from the end of the buffer, and (because of the way that
@code{regexp-search-backward} is implemented) the backwards regexp match
for @samp{\n*} is always the empty string!  The proper way to write this
is

@lisp
:cruft [[nil ["[^\n]\\(\n*\\'\\)" 1 "\n"]] [nil nil]]
@end lisp

@xref{Control properties reference}.


@node Tagged file layout
@section Tagged file layout
@cindex file layout, tagged
@cindex data file layout, tagged
@cindex tagged file layout

Another popular file layout supported by EDB is that of field values
preceded by the fieldname.  For instance, a record might be represented
in the file by

@example
Where:Here
When:  Now
What: This!
@end example

which indicates a record in which the `where', `when', and `here' fields
have the specified values.

Tagged files are a special case of files in nonregular layout;
EDB supports them specially (@pxref{Nonregular file layout}).

To read a database file in tagged format, call the function
@code{db-tagged-setup} in the database's format or auxiliary file.  Its
argument specifies the names of the fields and the tags that precede
them in the database file.  This function is called automatically when
the @code{:tagged-setup} control property is specified.

@defun db-tagged-setup fspecs [attrs...]
Ready the database to read files in tagged format.
Argument @var{fspecs} is a list of tagged-field specifications, one
for each field in a database record.  Each tagged-field specification is a
three-element list of the field name (a symbol), the tag used to identify
it in the file (a string), and a brief help string.
Instead of a symbol, the tagged-field name may be a cons of the
field name and its type.
To indicate that a field is never found in the input file (typically
because it is computed on the fly), use nil for its tag.
@var{attrs} is a sequence of alternating keywords and values specifying
overriding attributes.

Note: Do not call @code{database-set-fieldnames-to-list}
if you call this function.
@end defun

Calling @code{db-tagged-setup} sets the database's field names,
installs appropriate functions for reading and writing the database,
and allows you to set attributes specific to a tagged database.  For
example, you can customize the behavior of the parsing and output functions
with respect to what characters can appear in a tag, what the separator
between tag and value looks like, and how continuation lines are
handled.

By default, records are separated by blank lines, tags are
separated from field values by @samp{:}, white space around the
separator is not significant on input, the separator is followed by one
tab on output, and continuation lines start with whitespace.

Some attributes permit arbitrary manipulations of records; for instance,
if a database nearly conforms to the tagged file model, these can be used
to customize the behavior of the existing tagged code.  One way to do this
is to set @code{:pre-parse-thunk} to a function that removes the field
from the file representation before the record is parsed, and
@code{:post-write-function} to a function to modify the automatically
generated tagged file representation for that field.

When using the tagged file layout, you should not (due to implementation
restrictions) set variable @code{db-after-read-hooks} (@pxref{Read hooks}).
Instead, use the attribute @code{:index-function}.

The following keywords are recognized in as part of the @var{attrs} argument
to @code{db-tagged-setup}.  Unless otherwise specified, the default value
is @code{nil}.

@macro tskeyword{name}
@item \name\
@vindex \name\ db-tagged-setup keyword
@end macro

@table @code
@tskeyword{:tag-chars}
The characters that are allowed in field tags, in a string suitable for
placing inside @code{[]} in a regular expression.  Default value:
@code{-_A-Za-z}.

@tskeyword{:separator}
The string that separates field names from values.
Used only if @code{:separator-regexp}
or @code{:separator-output} is
@code{nil} (depending on whether the record is being read or written).
Default value: @code{:}.

@tskeyword{:separator-regexp}
A regexp for the separator between field names and values when parsing.

@tskeyword{:separator-output}
The separator between field names and values on output.

@tskeyword{:continuation}
The string that marks (the beginning of) a continuation line.
Used only if @code{:continuation-regexp}
or @code{:continuation-output}
is @code{nil} (depending on whether the record is being read or written).
Default value: @code{\t}.

@tskeyword{:continuation-regexp}
A regexp for a continuation line in a value when parsing.
Default value: @code{[ \t]+}.

@tskeyword{:continuation-output}
The fixed string to use (before) continuing values on output.

@tskeyword{:pre-parse-thunk}
On read, a function called with no arguments immediately prior to the
tagged parsing.  The buffer is narrowed to the region representing the
current record.

@tskeyword{:index-function}
On read, a function called with one arg @var{database}, after all the
records are parsed and converted from stored strings to actual objects.
This is useful for doing accounting chores, such as indexing, on the
entire set of records (@pxref{Mapping over the database}).

@tskeyword{:pre-write-function}
On write, called with one arg @var{record} before tagged writing.
Point is where the record will be inserted in the buffer.

@tskeyword{:post-write-function}
On write, called with one arg @var{record} after tagged writing.
Point is immediately after the file representation of the record.

@tskeyword{:default-field}
A fieldname (symbol) for the field indicated by an illegal or empty tag.
(For instance, you might set it to @code{comments}.)
If @code{nil}, those values are discarded.
@end table

@unmacro tskeyword


@node Relational file layout
@section Relational file layout
@cindex file layout, relational
@cindex data file layout, relational
@cindex relational file layout

[This node's documentation is poor primarily because, lacking examples
of db-rdb.el usage to study, we make uninformed guesses.  ---ttn]

The @file{db-rdb} library provides some functions for treating a
database in a manner that allows correlation between fields, also known
as relational capabilities, once the data is completely loaded into
memory.  On disk, the data must be stored in a variation of tagged file
layout (@pxref{Tagged file layout}).

To access this kind of database, use @code{db-rdb-setup}.  To correlate
fields, use @code{db-rdb-correlate-field-defs}.

@findex db-rdb-database-stored->actual
@findex db-rdb-list-rrfr
@findex db-rdb-list-wrfr
@findex db-rdb-read-fields
Other (possibly useful) functions:
@code{db-rdb-database-stored->actual}, @code{db-rdb-list-rrfr},
@code{db-rdb-list-wrfr}, and @code{db-rdb-read-fields}.

@defun db-rdb-setup rfspecs &optional lock-flag
Ready the database to read files in RDB format.  This creates database
local variables and sets database slots.  @var{rfspecs} is a list of
rdb-field specifications, one for each field in a database record.
Optional, second argument @var{lock-flag} should be non-@code{nil} to
lock the file for synchronized updates.  The locking and unlocking is
done with external programs @file{rdblock} and @file{rdbunlock}, which
must be available in the current PATH environment variable.

Each field specification is a three-element list of the field name (a
symbol), the tag used to identify it in the file (a string), and a
brief help string.  Instead of a symbol, the rdb-field name may be a
two-element list of the field name its type.  To indicate that a field
is never found in the input file (typically because it is computed on
the fly), use @code{nil} for its tag.
@end defun

@defun db-rdb-correlate-field-defs ifields efields
Correlate the @var{ifields} (internal fields) and @var{efields}
(external fields) with each other and produce a field list suitable for
@code{M-x db-rdb-setup}.

@var{ifields} is a list: @code{((@var{handle} @var{name}
@var{help})...)}, where @var{handle} is either @code{(@var{symbol}
. @var{type})} or just @var{symbol}.

@var{efields} is a list: @code{((@var{name} @var{width} @var{format}
@var{help})...)}, produced by the function @code{rdb-read-field-defs}.

If @var{type} is hidden, it is deduced from the corresponding
@var{format}.  Similarly, if the internal @var{help} is hidden, any
external @var{help} is used.  The internal definition always overrides
the external, since it is more specific to the EDB implementation.

The resulting list format is: @code{(((@var{symbol} . @var{type})
@var{name} @var{help})...)}.
@end defun


@node Nonregular file layout
@section Nonregular file layout
@cindex file layout, nonregular
@cindex data file layout, nonregular
@cindex nonregular file layout

Unlike most databases, EDB can work with data stored in any file layout
whatever --- so long as you specify how the information is to be
extracted.  If the file layout is too complicated to be described by
regular expressions describing the record and field separators and their
context (@pxref{Delimited file layout}), then you may write Emacs Lisp
code which extracts the information from the database file.

The great advantage of this mechanism is that it permits you to maintain
your current files, in exactly their current file layouts, and to keep
the same tools and habits you've accumulated, but also to manipulate
them in a structured way with EDB@.  For instance, you might wish to
maintain the database file in a file format easy for people to read all
the time, rather than having to create a report for that purpose.

Three pieces of information must be provided: how to find the extent of
a file record, how to read a file record, and how to write a file
record.  The third may be omitted if the database is only being read in
the custom file layout (and will be saved in some more tractable file
format).  If the second is provided (that is, the @code{:read-record}
control property is specified), then the file will
be assumed to be in a nonregular file layout and the function specified
is used to read the database, no matter what other information is
provided.

Information about how to separate one record from another within the
file is specified by the usual control properties (@pxref{Control properties
reference}).
In many cases, even if the file layout of the data is
nonregular, it is easiest to describe the record separator with a
string or a regexp.  For more details, @xref{Delimited file layout}.
You may also specify a @code{:record-separator-function} control property.
The function should take one argument, the end of the previous record
(@code{nil} the first time it's called), and return a pair whose car is
the end of the current record and whose cdr is the start of the next
record (@code{nil} if there is no next record in the file).

The function specified for the @code{:read-record} control property is
called with no arguments with the current buffer narrowed to a single
file record (that is, narrowed to the representation of a single
database record).  It should parse the database's file layout and
return a record initializer (@pxref{Kinds of control property values}).

The function specified for the @code{:write-record} control property
takes zero or more args.  If zero, the record's values are dynamically
bound to variables whose names are the field names.  If non-zero, the
function's arglist specifies those fields whose values are to be passed
in.  (This latter case is useful if some fields are computed.)  The
function should insert
the file representation of that record in the current buffer.

If @code{:write-record} is not specified (and the data is not in
internal layout format), then the field separator and
record separator information, if present, is used to write the record
(@pxref{Delimited file layout}).  This permits the use of a simple,
delimited output file layout with a more flexible input file layout.

Tagged format is a special case of nonregular file layout for which EDB
provides support (@pxref{Tagged file layout}).  Another
example is given below.


@menu
* Nonregular database example::
@end menu


@node Nonregular database example
@subsection Example of database in nonregular file layout

Here is a simple example of a database in a nonregular file layout; this
does not mean that the file representation of each record is vastly
different from the others (it may be, but is not in this instance), but
that there is no regular rule for extracting field values from the
record.

Suppose we had a database with fields @samp{place}, @samp{time}, and
@samp{purpose}, whose database file was:

@example
Dentist's Office at Never! for Root canal
Home at Midnight for Sleep
Other places at Other times for Other things
@end example

In order to read and write this database, place the following code in
the auxiliary file (@pxref{Reading from disk}):

@lisp
(setf (database-read-record-from-region database) 'arb-demo-rrfr
      (database-write-region-from-record database) 'arb-demo-wrfr)

(defun arb-demo-rrfr ()
  (unless (re-search-forward
           "\\(.*\\)\s-+at\\s-+\\(.*\\)\s-+for\\s-+\\(.*\\)")
    (error "This didn't look right to me."))
  (list 'place   (match-string 1)
        'time    (match-string 2)
        'purpose (match-string 3)))

(defun arb-demo-wrfr (record)
  (insert (db-record-field record 'place)
          " at " (db-record-field record 'time)
          " for " (db-record-field record 'purpose)))
@end lisp

The auxiliary file would also specify the database's field names:

@lisp
(database-set-fieldnames-to-list database '(place time purpose))
@end lisp

as well as possibly other information such as the summary format or the
name of the default format file.  See the example database auxiliary
file @file{arb-demo.dba} for a concrete example of this.

All this Emacs Lisp code may be placed in the local variables section of
the format file instead of in the auxiliary file, if desired.  For more
information about the local variables section of a file, @xref{File
Variables, , , emacs, The GNU Emacs Manual}.
@cindex local variables section, format file

This particular example is simple enough that a special function for
reading isn't strictly necessary.  Reading can be done under the control
of regular expressions; for instance, each field separator would be
@code{"\\s-+\\(at\\|for\\)\\s-+"}.  See the example database auxiliary
file @file{arb-demo-regexp.dba} for a concrete example of this.  You
would still need to specify a special record-writing function.

Here is another example, which has no field separators; in the data file,
the fields abut one another.  While it, too, could be read and written
under the control of regular expressions, the use of functions is a bit
clearer.

@cindex Einstein, Albert
@cindex Botswana
The data file is:

@example
Einsteinbirthday03141879
Botswanaindepend09301966
@end example

The auxiliary file contains the following:

@lisp
(setf (database-print-name database) "Historic Dates")

(database-set-fieldnames-to-list database
  '(name occasion month day year))

(setf (sepinfo-sep-string (database-record-sepinfo database)) "\n")

(setf (database-read-record-from-region database) 'sized-field-rrfr
      (database-write-region-from-record database) 'sized-field-wrfr)

(defvar sized-field-alist
  '((name . 8)
    (occasion . 8)
    (month . 2)
    (day . 2)
    (year . 4)))

(defun sized-field-rrfr ()
  (let ((field-begin (point)) pl)
    (dolist (this-size-cons sized-field-alist)
      (forward-char (cdr this-size-cons))
      (push (buffer-substring field-begin (point)) pl)
      (push (car this-size-cons) pl)
      (setq field-begin (point)))
    (unless (eobp)
      (error "Found extra characters in this record."))
    pl))

(defun sized-field-wrfr (record)
  (mapcar (lambda (this-size-cons)
            (let ((v (db-record-field record (car this-size-cons))))
              (if (not (= (length v) (cdr this-size-cons)))
                  (error "Field %s value %S has length %d (should be %d)."
                         (car this-size-cons) v
                         (length v) (cdr this-size-cons)))
              (insert v)))
          sized-field-alist))
@end lisp

This example is actually a bit too simple.  Some of the fields could be
made non-strings, field constraints should keep the fields the right
length, and @code{sized-field-alist} should be a database-local
variable.


@node Reading from disk
@section What happens when a database is read in from disk
@cindex reading from disk, details

In brief, the following happens after you execute @code{db-find-file}:

@enumerate
@item
If the database is already read in and its buffer has not been killed,
the buffer is simply selected.  No other work is done.

@item
Otherwise, the database file is inserted in a special buffer of its own.
If the database is in EDB internal file layout (that is, if an
identifying header is found), it is read in immediately.  Otherwise, a
new, empty database is created.  In either case the dynamic variable
@code{database} is bound; this makes it possible to refer to the
database in the auxiliary and format files (even before it has been read
in, if it is not in EDB internal file layout).

@item
The format file is found (@pxref{Auxiliary files}), and the data display
buffer is created.

@item
Perform
the rest of the work necessary for setting up the data display buffer
(everything up to the running of @code{db-before-read-hooks}).  The
first action is to insert the format file's contents into the data
display buffer.

@item
The auxiliary file, if any, is loaded.  This happens in the data display
buffer, and the dynamic variable @code{database} is bound to the current
database.  For
more information about how the auxiliary file is found and what it can
do, @xref{Auxiliary files}.

The auxiliary file is not read every time the previous step occurs,
only when a database's primary display format is read.  (The
primary display format is the one initially selected when a database is
first read in.)

@item
The local variables section, if any, of the format file is executed;
this may set variables and execute Emacs Lisp code, exactly analogously
to the auxiliary file.  EDB ignores the value of
@code{inhibit-local-variables} when evaluating this code.  This section
is then deleted from the working copy of the file, so that it does not
appear in the data display buffer when you view database records.
@xref{File Variables, , , emacs, The GNU Emacs Manual}, for
more information about the local variables section of a file.

@item
Database information is propagated; for instance, the names of the
database fields are known by now, and various other internal state is
initialized depending on this information, if necessary.

@item
The format file is parsed, and literal text and formatting directives
are distinguished from one another.
This completes the preparation of the data display buffer.

@item
Run @code{db-before-read-hooks}.

@item
If the database had already been read because it was stored in internal
file layout, it is massaged a bit to get it into its final form.

Otherwise, the database is finally read; the values of the
@code{:record-FOO} and @code{:field-separator} control properties determine
whether the layout is delimited or nonregular and direct the parsing.  The
@code{:substitutions} control property directs replacement of
characters that could not
be written into the file, and the @code{stored->actual} slot of each
record field type completes the translation to the data's internal format from
its file layout.

@item
Run @code{db-after-read-hooks}.

@item
The database has now been read and is in its final form.  The first
record of the database is displayed in the data display buffer, which is
then placed in view mode and selected (made visible).
@end enumerate


@node How information is displayed
@chapter How information is displayed
@cindex display format, specifying

The display of information, both on the screen (whether in the data
display buffer, the summary buffer, or elsewhere) and in other output
(such as reports), is controlled by formatting commands.  We will
discuss a data display buffer by way of example; the formatting
specifications are the same for summary buffers and reports as well.

Display types should not be confused with record field types; a display
type is used to specify how a particular value is shown on the screen,
but a record field type constrains the information actually contained in
the record field.  This chapter does not discuss record field
specifications, which specify everything about a record field type
except how it is displayed and parsed in output intended for humans to
read.  For more information about that, and about the distinction
between record field types and displaytypes (the latter of which is
described in this chapter), @pxref{Record field types}.

A display format gives all of the information necessary to create
a data display buffer; it consists of literal text that is displayed as
is (without possibility of editing) and of @dfn{display
specification}s that instruct EDB how to display a particular field's
contents.  The display specifications do not appear in the data display
buffer; they are replaced by fields' values, which may or may not be
editable.  An example of a display specification is
@samp{\name,width=16}, which indicates that the @samp{name} field of the
database should be displayed (after being padded or truncated to exactly
16 characters).

When a format is first specified, it is parsed and the formatting
information specified in the display specification strings is used to
create a displayspec structure.

@menu
* Display specifications::
* Predefined displaytypes::
* Enumeration displaytypes::
* Defining new displaytypes::
* Display specification optional parameters::
@end menu


@node Display specifications
@section Display specifications
@cindex display specification

A @dfn{display specification} describes how a particular database record
field appears in the data display buffer.
An display specification consists of a backslash followed by a field
name, plus perhaps some optional type and formatting information, plus
optionally a backslash followed by a space.  The items of extra
formatting information must be separated from the field name and from
each other by commas.  No spaces or tabs may occur in a display
specification.  To specify a backslash which does not begin a display
specification, but should appear in the data display buffer verbatim,
precede it by another backslash.

Here is a (quite complicated) example display format:

@example
\name,one-line-string,actual->display=upcase\ , \occupation'
Pay:     \\\salary,min-width=4:  too much!
Address: \address,indent is home sweet home
@end example

This display format is valid if the database contains fields called
@samp{name}, @samp{occupation}, @samp{salary}, and @samp{address}; any
other fields are not displayed.  Some typical records would be displayed
like this:

@example
JOHN DOE, butcher
Pay:     \  22:  too much!
Address: 123 Main St.
         Anyplace, USA is home sweet home
@end example

@example
JANE ROE, baker
Pay:     \4444444:  too much!
Address: 675 Massachusetts Avenue is home sweet home
@end example

The optional information includes the type of this display field and
formatting directives for it; if the type is present, then it must come
first among the displayspec's optional specifications.  Each optional
parameter is preceded by a comma to separate it from the preceding one
(or from the fieldname, for the first optional parameter).  The optional
information is typically of the form @samp{@var{slotname}=@var{value}},
which sets the specified slot to the given value, or
@samp{@var{slotsetter}}, which sets some slot to a particular value.
Explicitly specified formatting information overrides any defaults.  For
a list of slotnames and slotsetters, @pxref{Display specification
optional parameters}.

The display type can be specified by writing the typename (such as
@samp{string} or @samp{integer}) as the first optional parameter.

@cindex displaytype, not set by display specification
The display type specifies default values for the display specification
(actually for the displayspec structure, which is derived from the
display specification).  It is rarely necessary even to specify the
displaytype --- most display specifications consist of simply a backslash
and a fieldname --- since if the displaytype is omitted then a displaytype
with the same name as the record field type (actually the @code{:type}
attribute of the record field type) is used.  This works because typically
displaytypes and recordfieldtypes with the same names and complementary
definitions are declared at the same time.  The displaytype must be
compatible with the record field type; it is an error to specify a
displaytype of @samp{integer} when the data is actually a string.
@code{database-set-fieldnames-to-list} to specify recordfieldtypes;
@pxref{Specifying a record field type}.


@node Predefined displaytypes
@section Predefined displaytypes
@cindex displaytypes, predefined
@cindex type, display, predefined
@cindex predefined displaytypes

The following displaytypes are predefined.  Most of them correspond
directly with the same-named record field type; @xref{Predefined record
field types}.  You can also define displaytypes of your own;
@xref{Defining new displaytypes}.

@macro dtype{name}
@item \name\
@cindex \name\ displaytype
@end macro

@macro dtypex{name}
@itemx \name\
@cindex \name\ displaytype
@end macro

@subsection Builtin

These displaytypes are available as soon as EDB is loaded.

@table @code
@dtype integer
Ordinary integers.

@dtype integer-or-nil
Integers or @code{nil}, the empty value; @code{nil} is formatted as the
empty string.

@dtype number
@cindex floating-point number displaytype
Ordinary numbers.  A number is an integer or a floating-point number.

@dtype number-or-nil
Numbers or @code{nil}, the empty value; @code{nil} is formatted as the
empty string.

@dtype yes-no
This displayspec corresponds to the @code{boolean} recordfieldtype.
The field is three characters long and contains ``Yes'' or @w{``No ''}.

@dtype string
Ordinary strings.  By default there is no maximum or minimum width or
height, and subsequent lines are indented relative to the first
character of the first line.

@dtype one-line-string
Strings which may not contain newlines.

@dtype string-or-nil
Either a string or the value @code{nil}, which is displayed as the empty
string.

@dtype nil-or-string
Either a string or the value @code{nil}.  When you enter the empty
string as the field value, or when a new record is created, the value
@code{nil} is used in preference to the empty string.

@dtype one-line-string-or-nil
Either the value @code{nil} or a string which may not contain newlines.
@end table

@subsection From @file{edb-t-timedate1.el}

Loading @file{edb-t-timedate1.el} provides additional displaytypes,
and related functions.

@table @code
@dtype date
@findex edb-t-timedate1:date-year
@findex edb-t-timedate1:date-month
@findex edb-t-timedate1:date-day
A date which specifies zero or more of the year, month, and day; any or
all of these components may be hidden.  The date is created by the
constructor @code{edb-t-timedate1:make-date} and a date's components are
retrieved using the selectors @code{edb-t-timedate1:date-year},
@code{edb-t-timedate1:date-month}, and @code{edb-t-timedate1:date-day}.
A date is formatted by @code{edb-t-timedate1:format-date} and parsed by
@code{edb-t-timedate1:parse-date-string}.

@defun edb-t-timedate1:make-date year month day
Make an EDB date object representing @var{year}, @var{month},
and day @var{day} (all integers).
@end defun

@findex edb-t-timedate1:parse-date-default-function
@defun edb-t-timedate1:parse-date-string s
Parse string @var{s}, and return an EDB 1.x date object.  Signal error
if the parse is invalid.  If @var{s} contains only whitespace, return a
null date object.  If @var{s} is @code{nil}, use the result of calling
@code{edb-t-timedate1:parse-date-default-function} instead.
@end defun

@defun edb-t-timedate1:format-date fmtstr &optional date
Using @var{fmtstr}, format the @var{date}, which defaults to the current
date if @code{nil}.  @var{fmtstr} can contain the following symbol
strings, which are substituted by their corresponding value from the
date; other characters are inserted as-is.

@table @samp
@item %d
day of month --- 1 to 31 (one or two digits)
@item %dd
day of month --- 01 to 31 (always two digits)
@item %m
month of year - 1 to 12 (one or two digits)
@item %mm
month of year - 01 to 12 (always two digits)
@item %mon
month name (abbreviated) - Jun
@item %month
full month name - June
@item %yy
last 2 digits of year - 00 to 99
@item %year
year as 4 digits --- 0000 to 9999
@item %jday
Julian day of year --- 1 to 366
@item %wday
day of week --- 0 to 6 (Sunday = 0)
@item %day
day of week name --- ``Sun'' to ``Sat''
@item %weekday
full day of week name --- ``Sunday'' to ``Saturday''
@end table

See the variables @code{edb-t-timedate1:format-date-sub-syms-alist} and
@code{edb-t-timedate1:format-date-sub-syms-regexp}.

A special case: if an element of @var{date} is @code{nil}, its field is
hidden.  A @var{date} object of all @code{nil}s is thus formatted as the
empty string.
@end defun

@defun edb-t-timedate1:simple-format-date date
Format the @var{date} using a default format, defined by the variable
@code{edb-t-timedate1:simple-format-date-default}.  If @var{date} is
@code{nil}, use the value of @code{edb-t-timedate1:parse-date-default-function}.
@end defun

@item date-mmddyy "/"
@cindex date-mmddyy displaytype
@itemx date-yymmdd "" (empty string)
@cindex date-yymmdd displaytype
@itemx date-ddmmyy "."
@cindex date-ddmmyy displaytype
@itemx date-ddmmmyy " " (space)
@cindex date-ddmmmyy displaytype
@itemx date-yyyymmdd "/"
@cindex date-yyyymmdd displaytype
These are based on the @code{date} displaytype, and
are meant to be used in a display spec, such as:

@example
\datefield,date-mmddyy
@end example

The formatting for
each @code{date-@var{foo}} is done by a function
@code{edb-t-timedate1:format-date-@var{foo}}, for example
@code{date-mmddyy} and @code{edb-t-timedate1:format-date-mmddyy}.
These functions all accept an optional second arg @var{separator}
that can be used to override the default (shown after the name above).

@dtype date-iso
@dtypex date-europe
@dtypex date-full
@dtypex date-all
@dtypex date-unix
@dtypex date-dec
More @code{date}-based displaytypes, similar to the previous group,
except that the associated functions take only one arg (the date).

@dtype time
@findex edb-t-timedate1:format-time
@findex edb-t-timedate1:parse-time-string
A time which specifies zero or more of the hour, minute, and second.
The time is formatted by @code{edb-t-timedate1:format-time} and parsed by
@code{edb-t-timedate1:parse-time-string}.  The time is created by the
constructor @code{edb-t-timedate1:make-time} and a time's components are
retrieved using the selectors @code{edb-t-timedate1:time-hours},
@code{edb-t-timedate1:time-mins}, and @code{edb-t-timedate1:time-secs}.
A time is formatted by a @code{edb-t-timedate1:format-time-@var{spec}}
function and parsed by @code{edb-t-timedate1:parse-time-string}.

@defun edb-t-timedate1:make-time hours mins secs
Make an EDB time object representing @var{hours}, @var{mins},
and day @var{secs} (all integers).
@end defun

@findex edb-t-timedate1:parse-time-default-function
@defun edb-t-timedate1:parse-time-string s
Parse the first occurrence of @code{hh:mm:ss} in string @var{s}; return
a time object.  If ``:ss'' is hidden in @var{s}, the seconds default to
zero.  If @var{s} contains only whitespace, return an empty time object.
If @var{s} is @code{nil}, use instead the result of
@code{edb-t-timedate1:parse-time-default-function}.
@end defun

@defun edb-t-timedate1:format-time-12 time
@defunx edb-t-timedate1:format-time-24 time
@defunx edb-t-timedate1:format-time-hhmm time
@defunx edb-t-timedate1:format-time-12-hhmm time
@defunx edb-t-timedate1:format-time-24-hhmm time
Format @var{time} in various ways.
@end defun
@end table

@unmacro dtypex
@unmacro dtype


@node Enumeration displaytypes
@section Enumeration types
@cindex enumeration types

An enumeration displaytype is used for fields whose values are one of a
fixed set of alternatives.  Each alternative consists of an entire string
entered in the minibuffer with completion.  (The string may consist of
only a single character, but you must still type @key{RET}
after entering the string.  Also, @key{TAB} completes a partly-entered choice,
and @key{?} lists the remaining possibilities.
@xref{Completion, , , emacs, The GNU Emacs Manual},
for more about completion.)

The internal representation of the data --- its recordtype --- need have
nothing to do with the way that the alternatives are specified.  This section
describes the enumeration displaytype, which is nicknamed @dfn{enum}.

The internal,
input, display, and file storage representations of the value may all be
different.  This displaytype is
created by calling the following function, which also
creates a corresponding recordfieldtype.

@defun edb-define-enumtype typename alternatives [optstring]
Make @var{typename} (a symbol or string) an enumerated type.
Both a displaytype and a recordfieldtype are created.

@var{alternatives} is a list.  Each alternative is a list of up to four
components:
 the internal representation, any constant Lisp object, often a string;
 the input representation input interactively to specify this alternative,
   a string or list of strings (for multiple input representations);
 the display representation, a string; and
 the file storage representation, a string.

If the input representation is omitted and the internal representation is a
string, that string is used.  If the display representation is omitted, it
defaults to the first input representation.  The display representation is
automatically also a valid input representation.  If the file storage
representation is omitted, it defaults to the display representation.
If all the other components are omitted, the internal representation string
may be used in place of a one-element list containing just it.

Optional argument @var{optstring} is a displayspec option string.
@end defun

When a record field's type is an enum type, it can be
assumed that the value in the record field is
one of the valid representations.  (Similarly, when a field's type is
string, EDB can assume that the field content is actually a string.)
This means that the empty string, @code{nil}, and other special values
must be specifically mentioned when the enumeration type is defined.
Here is a way to define an enumeration type which is either a day of the
week or the empty string:

@lisp
(edb-define-enumtype 'workday
  '("Monday" "Tuesday" "Wednesday" "Thursday" "Friday" ""))
@end lisp

If it is possible for the field value to be @code{nil} (but not the
empty string) after reading the database, and @code{nil} should be
displayed as @samp{Unknown} (and that string parsed into a value of
@code{nil}), the following definition suffices:

@lisp
(edb-define-enumtype 'workday
  '("Monday" "Tuesday" "Wednesday" "Thursday" "Friday" (nil "Unknown")))
@end lisp


@node Defining new displaytypes
@section Defining new displaytypes
@cindex displaytype, defining
@cindex type, display, defining
@cindex defining displaytypes

When you are about to type a complicated display specification --- or a
simple one more than once --- consider defining and using a displaytype
instead.  Displaytypes are more concise (and so less cumbersome and less
error-prone), easier to change (since a change to the displaytype can
affect every display specification that uses it), and clearer (since a
descriptive typename makes immediately clear what the intention is).
Furthermore, displaytypes can be built up incrementally, with each one
making a few changes to those from which it inherits defaults.

Use @code{edb-define-displaytype} to define a new displaytype.

@defun edb-define-displaytype name source [override...]
Define a displaytype @var{name} (a symbol) with @var{source}.
@var{source} may be a displayspec, the name (a symbol) of a currently
defined type, a string representing the optional parameters part
of a displayspec, or nil.  In all cases, a new displayspec object
is created and then modified by @var{override}, a sequence of alternating
keywords and values, and finally added to a global list.  Return @var{name}.
@end defun


@node Display specification optional parameters
@section Display specification optional parameters
@cindex display specification optional parameters
@cindex displayspec fields

This section describes the display specification optional parameters,
which correspond exactly to slots of the displayspec, EDB's
internal representation of the display specification.

@cindex slotsetters, for display specifications
@cindex slot assigners, for display specifications
Optional display specification parameters are separated only by commas;
display specifications never contain whitespace.  These parameters are
of two forms: slotsetters, which are a single word and set a slot to a
particular value; and slot assigners, which are of the form
@samp{@var{slotname}=@var{value}} and set the slot to the value.  Unless
otherwise specified, each slot can be set by a slot assigner whose name
is the same as that of the slot.  An example of a display specification
containing two optional parameters, one a slot assigner and one a
slotsetter, is @samp{\name,width=16,unreachable}.

Display specification fields are processed in order, so only the last
instance of a particular parameter has any effect.  Any explicitly
specified parameter overrides defaults, values inferred from the type,
or previously specified parameters.

If you find yourself repeatedly writing similar display specifications,
or large, bulky display specifications, consider defining a new type to
do some or all of the work for you; @xref{Defining new displaytypes}.

@cindex displayspec structure

@macro dslot{name}
@item \name\
@cindex @code{\name\} displayspec slot
@end macro

@macro dslotx{name}
@itemx \name\
@cindex @code{\name\} displayspec slot
@end macro

@table @code
@dslot record-index
@cindex record field index
This integer is the field index in a database record of the value
formatted by this displayspec.  This is set by looking up the fieldname
part of the displayspec.

@dslot indent
@cindex @code{noindent} display specification parameter
This boolean value determines whether the second and subsequent lines
should align with the beginning of the first one or should be flush
left, in column 0.  It is set and unset with the @code{indent} and
@code{noindent} slotsetters.  The first of the following displays has
indent set, and the second does not:

@example
Name:    John Doe
Address: 123 Main St.
         Anyplace, USA
@end example

@example
Name:    John Doe
Address: 123 Main St.
Anyplace, USA
@end example

This causes alignment of the first character of subsequent lines with
the first character of the first line; it does not do anything clever
with whitespace in the field value, nor does it align different lines
differently.

@dslot min-width
@dslotx max-width
@cindex @code{width} display specification parameter
These integers are the minimum and maximum widths which the display may
occupy.  If the formatted value is too short, the function in the
@code{padding-action} slot is called to lengthen and/or justify it.  If
the formatted value is too long, the function in the
@code{truncation-display-action} slot is called to shorten it; if that
slot is empty, the field is simply truncated.  The @code{width} slot
assigner sets both the @code{min-width} and @code{max-width} displayspec
slots.  The @code{min-length}, @code{max-length}, and @code{length} slot
specifiers are synonyms for the @code{min-width}, @code{max-width}, and
@code{width} slot specifiers.

@dslot min-height
@dslotx max-height
@cindex @code{height} display specification parameter
These are analogous to @code{min-width} and @code{max-width}, but for
the number of lines occupied by the formatted value (actually, the
number of newlines in the string, plus one).  There is a @code{height}
slot assigner which sets them both.

@dslot truncation-display-action
@cindex @code{trunc-display} display specification parameter
This function helps reduce the size of the formatted value when it is
too large to fit in the specified displayspec size.  It defaults to
simply truncating the formatted field to the maximum permissible size.
It may also be set with the @code{trunc-display} slot assigner.
At present, this function is only called if the formatted value is too
wide; there is no analogous function called when it is too tall.

@dslot padding-action
@cindex justification of display fields
@cindex left justification of display fields
@cindex right justification of display fields
@cindex @code{right-justify} display specification parameter
@findex right-justify
This function determines how a field that is too small for the
displayspec (that is, the printed representation contains fewer
characters than specified in the @code{min-width} slot) should be
expanded to fit.  The padding function takes three arguments: the
minimum length, the unpadded display representation, and the length of
that representation.

The @code{padding-action} slot may also be set to a cons of a padding
character and a padding direction: @code{nil} for left-justification
(padding on the right), and non-@code{nil} for right-justification.
(You cannot set the @code{padding-action} displayspec slot to a cons by
using a display specification, since display specifications may not
contain whitespace, so the easiest way to right-justify a single field
is to use the @code{right-justify} slotsetter.)  The default, which can
also be obtained just by setting the slot to @code{nil}, is to pad on
the right with space characters.

@dslot actual->display
@dslotx display->actual
@cindex @code{a->d} display specification parameter
@cindex @code{d->a} display specification parameter
These functions convert between the data's
internal representation and its displayed representation (a string).
Other functions (such as those in the @code{truncation-display-action}
and @code{padding-action} slots) may then be called on the result
returned by the @code{actual->display} function.  These slots may be set
with the @code{a->d} and @code{d->a} slot assigners.

The @code{display->actual} function takes either one argument or four
arguments: either just the field text or the field text, the previous
field value, the record being operated upon, and the record fieldnumber
of the field in question.  EDB ascertains at runtime how many arguments
the function should be applied to.  The old field value is passed in
case it contains hidden (undisplayed) attributes that need to be
preserved across changes.  The other two arguments permit a particular
@code{display->actual} function to be used for more than one field of a
record, allow the field text parse to depend on other record field
values, and provide for other complicated needs.  Most
@code{display->actual} functions can be specified to take a single
argument.

The @code{actual->display} function takes either one argument or three
arguments: either just the field value or the field value, the record,
and the record fieldnumber.  EDB ascertains at runtime how many
arguments the function should be applied to.  The reasons the additional
arguments may be specified are similar to those outlined above (for
instance, to permit the displayed representation of a field to depend on
other information in the record); most @code{actual->display} functions
take just one argument --- for instance, @code{upcase} is a valid
@code{actual->display} function.

@dslot match-actual->display
@dslotx match-display->actual
These functions are like @code{actual->display} and
@code{display->actual}, but are only invoked when reading a displaying a
search specification.  If they are not specified (as is usually the
case), then the ordinary (@code{match-}-less) versions are used for
search specifications as well.

These slots should be set to symbols, not to functions proper; that is,
to specify that function @code{foo} should be used, set the slot to
@code{'foo}, not to @code{(function foo)}.

@c mernst sez: I don't know that these belong in the displayspec,
@c             but I don't quite know where they do belong.

@dslot truncation-editing-action
@cindex @code{trunc-edit} display specification parameter
This function specifies what to do when a field being edited is too
large for the specified displayspec size; this action may be different
from that taken when simply displaying the offending value.  It may also
be set with the @code{trunc-edit} slot assigner.

@dslot reachablep
@cindex uneditable fields in data display buffer
@cindex @code{unreachable} display specification parameter
A Boolean value determining whether movement commands should skip this
display field.  The @code{reachable} and @code{unreachable} slotsetters
are used to assign a value to this slot.

@end table

@unmacro dslot
@unmacro dslotx


@node Customization
@chapter Customization
@cindex customization

@menu
* Auxiliary files::
* Hooks and customization functions::
* Global variables::
@end menu


@node Auxiliary files
@section Auxiliary and format files
@cindex auxiliary file
@cindex format file
@cindex display format file name
@cindex file name, for auxiliary file
@cindex file name, for display format

When using control properties, this section has no meaning (you can skip it),
since aux file functionality is handled by the free-standing Lisp in the
control file.  EDB 2.x will not have handling for format and auxiliary
information in separate files outside that provided by the @code{:display}
control property and @code{load}.

You can provide code to be
executed when the database is read in (@pxref{Reading from disk}).  The
optional auxiliary file usually contains the code specific to a
particular database, but the format file, which specifies the on-screen
arrangement of fields of a record, can also contain such code.

Since the auxiliary file is read after the format file has been found
but before it has been parsed, neither file can specify the other.  The
format file can, however, load arbitrary files, which is nearly as good
as being able to specify an auxiliary file.

@cindex auxiliary file name
EDB looks for a file with the same name as the database file, but
ending with one of the suffixes in @code{db-aux-file-suffixes}.

@defvar db-aux-file-suffixes
List of auxiliary file suffixes; the basename is that of the database file.
The suffixes are tried in order; the default is @code{(".dba" ".aux" "a")}.
The @code{.} that may precede the extension must be specified explicitly.
@end defvar

@defvar db-aux-file-path
List of directories (strings) to search, in order, for auxiliary files not
found in the directory with their associated databases.
@end defvar

The auxiliary file is evaluated in the data display buffer and so can
set variables local to that buffer, such as hooks (@pxref{Hooks and
customization functions}).  The database itself can be manipulated via
the dynamic variable @code{database}.
For instance, auxiliary files often set the
name (a string) of their associated databases.

Code in an auxiliary file should be specific to the particular database;
more general code is best placed in a separate file which is
@code{load}ed (or, better, @code{require}d) by the auxiliary file.  For
instance, if you want to permit EDB to manipulate files of type Foo, you
should put all Emacs Lisp code that applies to every Foo file in one
file (@file{db-foo.el}, say), and then put @code{(require 'db-foo)} in
the auxiliary file associated with a particular Foo file.  (Alternately,
you may autoload a function that will be called in the auxiliary file;
function @code{db-tagged-setup} is autoloaded from @file{db-tagged.el}
in this manner.)  Either technique keeps auxiliary files simple and small
and makes Foo-specific code easier to debug, byte-compile, and load only
once per session.  These advantages easily outweigh the introduction of
an extra file.

Since the format file has not yet been interpreted, the auxiliary file
could even change the contents of the buffer (and so the apparent
contents of the format file); such extreme trickiness is only called for
in special circumstances.

@cindex format file name
The format file can contain Emacs Lisp code in its local
variables section; that code can do anything that code in the
auxiliary file can do.  EDB
tries to find one based on the database file
name and the suffixes in @code{db-format-file-suffixes}; if that doesn't
work, you are prompted for a display format to use.

@cindex format file, path to search
@defvar db-format-file-suffixes
List of format file suffixes; the basename is that of the database file.
The suffixes are tried in order; the default is @code{(".dbf" ".fmt" "f")}.
The @code{.} that may precede the extension must be specified explicitly.
@end defvar

@defvar db-format-file-path
List of directories (strings) to search, in order, for format files not
found in the directory with their associated databases.
@end defvar

Code in the format file is useful for customizations specific to a
particular format (such as setting variables which are local to the
data display buffer); they can also be used for database-specific
customizations if the file will
always be the primary (first-selected) format for the database.


@node Hooks and customization functions
@section Hooks and customization functions
@cindex hooks
@cindex customization functions

The following sections describe EDB's hooks and customization functions.
@xref{Hooks, , , emacs, The GNU Emacs Manual}.

Many of these hooks are change hooks, which permit a function (or
functions) to be run whenever a value changes.  These change hooks may
be divided into two basic types: format change hooks and
record field type change hooks (however, the latter is not yet
implemented and probably will never be).  The former are associated with a
particular display format and are invoked when the value in a particular
field, or in any field, changes.  The latter (which are not yet
implemented) are associated with a record field type and are invoked
whenever a database record slot of a particular type is changed.


@menu
* Read hooks::
* Database mode hooks::
* Record display hooks::
* Edit mode hooks::
* Display format change hooks::
@end menu


@node Read hooks
@subsection Read hooks

The following two hooks are useful for causing database values seen by
EDB to be different than those in the database file.  The first can be
used to modify the database file before it is read in; the second can
be used to modify the database after it has been read in but before
EDB displays it.

@defvar db-before-read-hooks
Normal hook run immediately before a database is first read
but after all local variables are set.
The hooks are run in the data display buffer
with variable @code{database} bound.
Variable @code{db-buffer} is bound to a buffer containing the database file.

This is a global variable.  If you set it to be specific to a particular
database (for instance, in the format or auxiliary file), then consider
having its last action be to reset the variable to @code{nil}.
[Lame. ---ttn]
@end defvar

@defvar db-after-read-hooks
Normal hook run after a database is completely read.
The hooks are run in the data display buffer
with variable @code{database} bound.
For databases with nonregular layouts, you might put a call to
@findex database-stored->actual
@code{database-stored->actual} here, for instance.

This is a global variable.  If you set it to be specific to a particular
database (for instance, in the format or auxiliary file), then consider
having its last action be to reset the variable to @code{nil}.
[Lame. ---ttn]
@end defvar


@node Database mode hooks
@subsection Database mode hooks
@cindex database mode hooks
@cindex hooks, database modes

EDB provides hooks that are run whenever the data display buffer is
switched between Database View mode and Database Edit mode
and which are run when a
summary buffer is created.

@defvar db-view-mode-hooks
Normal hook run when Database View mode is entered.
@end defvar

@defvar db-edit-mode-hooks
Normal hook run when Database Edit mode is entered.
@end defvar

@defvar database-summary-mode-hooks
Normal hook run when switching to Database Summary mode.
@end defvar

When using control properties, you should use @code{:locals} variables
with the same name instead of setting the Emacs Lisp variables either
directly or with @code{add-hook}.  For example, here is a @file{.edb}
file fragment:

@lisp
(defun set-up-1 () (setq truncate-lines t))
(defun set-up-2 () (scroll-bar-mode 1))

:locals
[
 (database-summary-mode-hooks '(set-up-1 set-up-2))
]
@end lisp

@noindent
This arranges for the summary buffer to have a scroll bar and for its
long lines to extend off screen.  Note the square braces.

@node Record display hooks
@subsection Record display hooks
@cindex record display hooks
@cindex hooks, record display
@cindex display format, data-dependent
@cindex data-dependent display format

The following function is run each time a record is
about to be displayed.
This variable is set automatically when either of the control properties
@code{:before-display} or @code{:choose-display} is specified
(@pxref{Control properties reference}).

@defvar dbf-before-display-record-function
A function called before a record is displayed.
The function takes one argument, the record.

This is a good place to put calls to @code{db-change-format}.
@end defvar

Here is an example of how you might use this:

@lisp
(defun set-format-from-data (record)
  (if (< 0 (db-record-field record 'net-profit))
      (db-change-format "loss format" "~/acct/db/loss.fmt")
    (db-change-format "profit format" "~/acct/db/profit.fmt")))

(setq dbf-before-display-record-function 'set-format-from-data)
@end lisp

This uses two different display formats, depending on the value of one
field of a record.  As you move from record to record in the database,
each one is shown using the appropriate display format.  A preferable
implementation hides the filenames from the calls to
@code{db-change-format} and instead uses, in the format or auxiliary
file,

@lisp
(setq dbf-format-name-spec-alist
      '(("loss format" . "~/acct/db/loss.fmt")
        ("profit format" . "~/acct/db/profit.fmt")))
@end lisp

See the example file @file{arb-demo.dba} for an example of this.
@code{dbf-format-name-spec-alist} need not specify the full pathnames if
the format files are located in the same directory as the database or if
@samp{"~/acct/db"} is placed in @code{db-format-file-path}.

Finally, you would probably set a
change hook on the net-profit field so that when its value changed, the
record could be redisplayed in the appropriate format automatically.

Here is an example of how to change the data display format in
response to a particular field being changed.

@lisp
(defun equip-dbf-from-field (fieldname oldval newval)
  (let ((dbf-elt (assoc newval dbf-format-name-spec-alist))
        (save-index dbf-this-field-index))
    (db-change-format (if dbf-elt newval "default"))
    (setq dbf-this-field-index save-index) ; otherwise aref on nil
    ;; redisplay record
    t))

(setq db-after-read-hooks
      (lambda () (dbf-set-change-function
                  'equip-type
                  'equip-dbf-from-field)))
@end lisp


@node Edit mode hooks
@subsection Edit mode hooks
@cindex edit mode hooks
@cindex hooks, edit mode

These hooks are called whenever you enter a field to edit it, which
provides an easy way to customize the behavior of particular format
fields.

@defvar dbf-enter-field-hook
Normal hook run whenever a display field is entered.
The displayspec index is @code{dbf-this-field-index}.
@end defvar

Note: For a @code{.edb} file (when using @code{edb-interact}),
use the technique described in @ref{Database mode hooks}.

It is sometimes advantageous to have a particular action happen only
once per edit of a record.  For instance, when a record's address, city,
state, or zip-code fields are edited, we might like to copy all the
values to the old-address, old-city, old-state, and old-zip-code fields.
We only want this to happen once, however: if you edit first the
address, then the city, we don't want to repeat the process, because
then the old-address field would get written over by the new value of
the address field.

One way to prevent this from happening more than once is to set a
variable when the copying is done, and then don't do the copying if that
variable is set.  The variable would be reset whenever a new record was
edited.  You can use @code{db-edit-mode-hooks} to do this job.

@lisp
(setq db-after-read-hooks
      (lambda ()
        (set (make-local-variable 'db-edit-mode-hooks)
             (lambda ()
               (dolist (var '(tep-homeaddr-oldified
                              tep-homephone-oldified
                              tep-bizaddr-oldified
                              tep-bizphone-oldified))
                 (set (make-local-variable var) nil))))
        (dbf-set-change-function
         'home-phone 'tep-homephone-change-hook)
        (setq db-after-read-hooks nil)))

(defvar tep-homephone-oldified nil)

(defun tep-homephone-change-hook (fieldname oldval newval)
  (unless tep-homephone-oldified
    (dbf-this-record-set-field 'old-home-phone oldval))
  ;; non-nil return value means redisplay whole record
  (prog1 (not tep-homephone-oldified)
    (setq tep-homephone-oldified t)))
@end lisp


@node Display format change hooks
@subsection Display format change hooks
@cindex display format change hooks
@cindex change hooks, for display formats
@cindex hooks, change, for display formats

The following hook is run whenever a new record is created.
This hook is set automatically when the control property
@code{:record-defaults} is specified
(@pxref{Control properties reference}).

@defvar db-new-record-function
Function called on empty records before they're inserted in the database.
Takes two arguments, the record and the database.
@end defvar

@cindex new records, setting default information
A typical use is to set default information or add a timestamp.  For
instance:

@lisp
(defun set-update-date (record database)
  "Provide defaults for new records in the database."
  (db-record-set-field
   record 'updatedate
   (edb-t-timedate1:parse-date-string (current-date))
   database))

(setq db-new-record-function 'set-update-date)
@end lisp

The display format change hooks are called when you change a record
field value.  There are separate change hooks that run the first time
any field is modified, whenever any field is modified, and whenever a
particular field is modified.  They run in the order
@code{dbf-first-change-function}, @code{dbf-every-change-function}, and
finally one of the elements set by @code{dbf-set-change-function}.
Each change
hook is either @code{nil} or a function of three variables: the
fieldname of the just-modified field (a symbol) and the pre- and
post-modification field values.  These functions can set variable
@code{dbf-redisplay-entire-record-p} to a non-@code{nil} value in order
to cause the entire record to be redisplayed (for instance, if the
change hook modifies fields other than that named by its first
argument).
@vindex dbf-redisplay-entire-record-p

@defvar dbf-first-change-function
A function called the first time a record field is modified, or @code{nil}.
The function takes the fieldname and the old and new values as arguments,
and returns @code{t} if the record should be redisplayed.

This variable is set automatically when the control property
@code{:first-change-function} is specified
(@pxref{Control properties reference}).
@end defvar

@cindex last modification field
Here is an example of code to update the last modification field of a
record, assuming its type is date:

@lisp
(defun update-last-modified-date (fieldname oldval newval)
  "Put the current date in this record's `last modified' field."
  (dbf-this-record-set-field
   'last-modified
   (edb-t-timedate1:parse-date-string (current-date))))

(setq dbf-first-change-function 'update-last-modified-date)
@end lisp

@defvar dbf-every-change-function
A function called whenever a record field is modified, or @code{nil}.
The function takes the fieldname and the old and new values as arguments,
and returns @code{t} if the record should be redisplayed.

This variable is set automatically when the control property
@code{:every-change-function} is specified
(@pxref{Control properties reference}).
@end defvar

@defun dbf-set-change-function fieldname function
Set the change function for @var{fieldname} to @var{function}
in the current database.
@var{function} takes the fieldname and the old and new values as arguments,
and returns @code{t} if the record should be redisplayed.
@end defun

@cindex dependent field values
It is easy to make a field's value dependent on that of another field.
For instance, suppose a salesman's commission should be 10% of the
selling price of an item, and both fields are of type number.
You could make the commission field unreachable
(@pxref{Display specification optional parameters}) and compute it
whenever the selling price field varies.  The latter operation could be
done as follows:

@lisp
(defun set-commission (fieldname oldval newval)
  (dbf-displayed-record-set-field 'commission (/ newval 10)))

(dbf-set-change-function 'selling-price 'set-commission)
@end lisp

There are two things to notice about this example.  First, we need not
set @code{dbf-redisplay-entire-record-p}, as
@code{dbf-displayed-record-set-field} does that automatically.  Second, if
we replaced @code{newval} by @code{(dbf-displayed-record-field
'selling-price)}, then the function
would work even if not called as a change function for selling-price.

You may modify records explicitly by calling
@code{dbf-displayed-record-set-field} (@pxref{Manipulating records});
when that is done, the following hook is invoked.  It is different from
the above functions in that they are called when you edit a field,
while it is called when Emacs Lisp code modifies a field (usually as a
result of some interactive command).

@defvar dbf-set-this-record-modified-function
A function called when the current record is marked as modified.
The function takes no arguments and its return value is ignored.
It is called after the original record values are copied
to the working record.
@end defvar

Another function is invoked when changes to a record are
committed --- that is, when changes to the record which is being displayed
are copied back into its original in the database.

@defvar dbf-after-record-change-function
Function called whenever changes to a record are recorded semi-permanently.
The function takes the record as an argument.
Its return value is ignored.
@end defvar

Note: To achieve the same effect as setting these two variables for a
@code{.edb} file (when using @code{edb-interact}), use the technique
described in @ref{Database mode hooks}.

@node Global variables
@section Global variables
@cindex customization, global variables

This section describes some customization variables which you
can use to understand and (if lucky) control EDB's behavior.

When a potentially slow computation is underway, EDB displays a message
in the echo area reporting how many records have been processed.  Use
the following variable to control how often this message is updated.

@defvar db-inform-interval
When doing a lengthy computation, display a progress message in
the echo area every this many records.  If @code{nil}, don't inform.
@end defvar

Note: For a @code{.edb} file (when using @code{edb-interact}),
use the technique described in @ref{Database mode hooks}.

@node Database representation
@chapter Database representation
@cindex database representation
@cindex representation of database

Perhaps the most important information about a database --- besides the
records it contains --- is the number of fields in each record, and the
type of each field.  As explained previously
(@pxref{Terminology and concepts}),
a database
consists of records with identical numbers of fields; each field has an
associated type such as string or integer.  Each field also has a name
which is used when extracting its value from the record.

Internally, records and metainfo about them (individually and in aggregate)
are represented by various vectors and hash tables; however, you
should never manipulate those structures directly, only through the functions
described in this chapter.

@defvar dbc-database
This is a buffer-local variable that contains the @dfn{current database}.
@end defvar

Given a record, it is not possible to determine which database (if any)
it belongs to.  The database-to-record connection is one-way.


@menu
* Mapping over the database::
* Manipulating records::
@end menu


@node Mapping over the database
@section Mapping over the database
@cindex mapping over the database
@cindex looping over the database

Mapping refers to applying a function to each record in the
database, or executing a piece of code for each record.

@defun db-maprecords func [database [hide [message [accumulate]]]]
Apply @var{func} to every record in the current database in order of ascending
index.  Optional second arg @var{db} specifies a database to use other than
the current one.
If optional third arg @var{hide} is non-@code{nil},
apply @var{func} only to
unhidden records.
If optional fourth arg @var{message} is non-@code{nil},
it should be a format string
containing one numeric (%d) specifier.  That message will be issued every
@code{db-inform-interval} records.
If optional fifth arg @var{accumulate} is non-@code{nil},
return a list of the results;
otherwise return @code{nil}.
@end defun

There is also the alias @code{maprecords} for old code.
(If you can help it, use @code{db-maprecords} instead.)

For instance, to sum, for all records, the values contained in field
@samp{summand} (of type @code{number}), you could use either of the
following forms:

@lisp
(let ((result 0))
  (db-maprecords
   (lambda (record)
     (setq result (+ result (db-record-field record 'summand)))))
  result)

(apply (function +)
       (db-maprecords
        (lambda (record) (db-record-field record 'summand))
        nil nil nil t))
@end lisp


@node Manipulating records
@section Manipulating records
@cindex record representation

A database consists of records, each of which has the same makeup:
corresponding fields in a database's records contain data of the same
type.  For instance, the fifth field of each record might contain an
address, and the seventh field, a date.  The particular addresses and
dates would would vary from record to record.  (Different databases will
contain records with different numbers and types of fields.)  Each field
has a name and a type, which specifies what sort of information can be
stored in the field; for more details about record field types,
@xref{Record field types}.

Records are represented internally as vectors, but should never be
operated on as such; use the abstractions described in this section.

@subsection Creating and copying records
@cindex creating records
@cindex copying records

@defun db-make-record database init
Return a @var{database}-specific record initialized with @var{init}.
@var{init} is either a list of alternating field names (symbols) and values,
or a list whose car is the keyword @code{:alist} and whose cdr is an alist
mapping field names to their values.
@end defun

When you create a new record by using @code{db-add-record}
(@pxref{Adding and removing records}), @code{db-new-record-function} is
invoked (@pxref{Display format change hooks}), the number of records in
the database is modified, and so forth.  @code{db-make-record}, on the
other hand, performs none of these housekeeping tasks.

@subsection Accessing record fields
@cindex accessing record fields
@cindex setting record fields
@cindex reading record fields
@cindex record fields, accessing
@cindex fields, accessing them in records
@cindex fields, setting them in records
@cindex fields, reading them in records

Ordinarily, record fields are accessed by specifying the name of the
desired field; the database must also be specified so that the
fieldname-to-fieldnumber correspondence can be determined.

@defun db-record-field record fieldname [database]
Return from @var{record} the field with name @var{fieldname}.
If @var{record} is t, use the ``current record''.  Optional third
argument @var{database} specifies a database other than the current one.
@end defun

@defun db-record-set-field record fieldname value [database [nocheck]]
Set, in @var{record}, field @var{fieldname} to @var{value}.
Fourth argument is @var{database}.
Check constraints first unless optional
fifth argument @var{nocheck} is non-@code{nil}.
@end defun

There are also special commands for manipulating the current
record --- that is, the one that appears in the data display buffer.
These functions require fewer arguments, flag that a redisplay of the
record is necessary, and automatically call
@code{dbf-set-this-record-modified-p}, which is essential if the changes
are to be copied back into the original record in the database from the
one that is being displayed.  (A copy is always displayed so that
changes can be undone.)

@defun dbf-displayed-record
Return the record currently displayed in this data display buffer.
This may either be the working copy, if the record has been marked as
modified, or the original otherwise.
@end defun

@defun dbf-displayed-record-field fieldname
Return the value of the field named @var{fieldname} from the displayed record.
@end defun

@defun dbf-displayed-record-set-field fieldname value
Set field with name @var{fieldname} in displayed record to @var{value}.
Cause the entire record to be redisplayed soon.
@end defun

@defun dbf-displayed-record-set-field-and-redisplay fieldname value
Set field with name @var{fieldname} in displayed record to @var{value}.
Cause the entire record to be redisplayed immediately.
@end defun

@defun dbf-set-this-record-modified-p arg
Mark the current record as modified if @var{arg} is non-@code{nil}.
If the record was previously not marked as modified, make a copy
of the original, and call @code{dbf-set-this-record-modified-function}.
@end defun


@node Naming conventions
@chapter Naming conventions

@cindex naming conventions
EDB makes use of several namespaces: Emacs lisp functions and variables, file
names, and type names.  To reduce confusion, names have been chosen following
some conventions, described in this chapter.

@section Function and variable naming conventions
@cindex function naming conventions
@cindex variable naming conventions

Functions and variables begin with one of the following prefixes.

@table @code
@item edb-t-GROUP:
Functions or variables supporting some @dfn{type group} (for
lack of a better name).  Note the trailing @kbd{:}.  The type
names themselves do not presently follow any conventions,
although that may change in the future with EDB 2.x.

@item edb-
Data structures and functions related to the (meta) binding behavior.
Many of these have two hyphens instead of one, which means the item
is @dfn{internal} in some sense; its definition may change without
warning from one EDB release to the next.

@item db-
In a variable, indicates that the variable is global and affects all
databases.  It is also used in some situations for
internal database functionality which is not connected with any
particular buffer.

@item database-
These functions operate on (the internal representation of) the database
structure itself.

@item dbc-
Indicates a variable local to the data display buffer which refers to
the current database (the database being manipulated by that data
display buffer).  The `c' stands for ``current.''

@item dbf-
Indicates a variable local to the data display buffer which controls
some aspect of formatting.  The `f' stands for ``format''; many such
variables are intimately related to the format, and the data display
buffer used to be called the format buffer.

@item dbs-
Indicates a variable local to the summary buffer, or a summary buffer
function.  Since the summary buffer may disappear at any time, the
summary buffer gets most of its information from the associated data
display buffer's local variables.

@item dbsi-
Indicates a variable local to a sort interface buffer, or a sort
interface function.
@end table

@section File naming conventions
@cindex file naming conventions

By convention, database file names contain one of the following suffixes:

@table @code
@item edb-t-GROUP.el
Contains forms that implement a type group (see above).

@item .dat
These are database files proper; they contain the information that makes
up the fields and records of the database.  Database filenames may also
contain no extension at all.

@item .fmt
Format files control the structure of the data display buffer, which
displays one record at a time.

@item .dba
Auxiliary files contain arbitrary Emacs Lisp code; they can be used to
define functions, set variables, or operate directly on the database.
@end table

Other suffixes --- or none at all --- can be easily used; for instance,
see variables @code{db-format-file-suffixes} and
@code{db-aux-file-suffixes} (@pxref{Auxiliary files}).

For more information, @xref{Invoking EDB}.

@section Type naming conventions
@cindex type naming conventions

There is only one minor naming convention for types: @code{BASE-or-nil}
usually means that handling for the type is very similar to handling for type
@var{base} with special handling in case the value is @code{nil}.


@node In case of trouble
@chapter In case of trouble

@section Variables
@cindex variable documentation missing
@cindex variable default value missing
@cindex trouble with undefined variables

In some cases the documentation strings and/or default values of some
variables may be missing --- as if the variables were not yet defined.
That's because the variables are not yet defined; they are associated
with part of EDB which hasn't been loaded because it hasn't been needed
yet.  The documentation and default values will appear when that part of
EDB is loaded (if you set such variables, your values will not be
replaced).  Such variables are correctly declared buffer-local (if
appropriate), so you can set them without fear of the changes affecting
other buffers.

@section Exiting Emacs or saving files
@cindex exiting Emacs, trouble with
@cindex trouble with exiting Emacs
@cindex saving files, trouble with
@cindex trouble with saving files

If you are unable to exit Emacs or to execute
@code{db-save-some-buffers} because Emacs is trying to
manipulate a database which doesn't exist or because an EDB bug is
triggered by the attempt to save an existing database, you can
set variable @code{edb--global-state} to nil and reload @file{edbcore.elc}.
[TODO: Write a reset command; document it here. --ttn]
This indicates to EDB that
there are no databases read into memory and, therefore, no operations
will be attempted on them as a part of saving all modified Emacs
buffers.
@findex db-save-some-buffers

@section Poking Around

If things get really weird, you can try to use the command @code{edb-meta}
and include the result in a bug report to the EDB maintainer.

@defun edb-meta
Summarize EDB state in a new buffer, and switch to it.  The output format is
likely to change from one release of EDB to the next; do not rely on it.
@end defun


@node Function Index
@unnumbered Function Index

@printindex fn

@node Variable Index
@unnumbered Variable Index

@printindex vr

@node Concept Index
@unnumbered Concept Index

@printindex cp

@c To prevent the Concept Index's last page from being numbered "i".
@page

@summarycontents
@contents
@bye
