This file documents freeWAIS-sf version 2.0.

Copyright 1995 University of Dortmund, Germany.

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the Free Software Foundation.

freeWAIS-sf
***********

This file documents freeWAIS-sf version 2.0.

Copyright © 1995 University of Dortmund, Germany.

Overview
********

The freeWAIS-sf distribution contains clients, servers, and auxiliary programs for the TCP/IP protocol known as WAIS. See Protocol. The distribution is focussed on the WAIS server waisserver and the indexer waisindex. Clients are distributed only for convenience. You can use any WAIS client to query the freeWAIS-sf server.

The WAIS Protocol
=================

The WAIS protocol is based on the ANSI standard Z39.50 Version 1 (aka Z39.50 V1 or Z39.50 88). It uses only a subset of features of the Z39.50. Namely no requests which require the server to preserve some state are used. The WAIS protocol extents this basic features with elements for relevance feedback. The WAIS protocol is sometimes slightly incorrect also referred to as Z39.50 V1.

This ad hoc extension found their way back to the Z39.50 standard when the next version Z39.50 V2 (aka Z39.50 92) was defined. For implementation of WAIS-like (remember: WAIS is the name of the original protocol) the exist a definition of subset called the "WAIS Profile". Often Z39.50 V2 is used as synonym for this profile. freeWAIS-sf currently does not support Z39.50 V2.

Client Server Architecture
==========================

A WAIS System consists of clients talking to a server via an TCP/IP network using the WAIS protocol. Servers answer search requesta using auxiliary data structures called the index. These are created for the original documents by the waisindex program. Retrieve-requests are answered by the server fetching (parts of) the original files.  As set of documents together with their associate index is called a WAIS database.

                                +------+
                                |Client|
                                +------+
                                    |
                                 TCP/IP
                                 network
                                    |
                                +------+
                          +---->|Server|<----+
                          |     +------+     |
                          |                  |
                       +-----+          +---------+
                       |Index|          |Documents|
                       +-----+          +---------+
                          ^                  |
                          |     +-------+    |
                          +-----|Indexer|<---+
                                +-------+

Getting freeWAIS-sf and related stuff
*************************************

Mirror Sites
============

Most of the ftp accessable Stuff is mirrored on different sites. Please check archie or one of the sites below. The load on our server is growing out of bounds. We had to close it for the public - for the big files at least. Since all Packets are routed over the EUnet site, it will be faster to get the stuff from there as from ls6-www.

ftp.wsct.wsc.com
     /pub/freeWAIS-sf/

ftp.sysadmin.com
     /pub/admin/www/misc/wais-sf/

ftp.maxwell.syr.edu
     /infosystems/wais/FreeWAIS-sf/

ftp.germany.eu.net
     /pub/infosystems/wais/Unido-LS6/

ftp.switch.ch
     /mirror/wais/servers/sf-alpha/

ftp.leo.org
     /pub/comp/infosys/wais/freeWAIS/

ftp.egg.com
     /pub/infosystems

If you want to become a mirror site for freeWAIS-sf, send a mail to wais@ls6.informatik.uni-dortmund containing:

  * The ip-address and ip-name of the host you mirror from.

  * The ip-name of the serveryou want to redistribute and the directory where the distribution will be available.

  * The frequency you will mirror the distribution.

Documentation
=============

freeWAIS-sf-faq
     The freeWAIS-sf FAQ. Information which did not found its way in this manual.

       * ftp://MIRROR-SITE/MIRROR-DIR/freeWAIS-sf-faq

       * http://www.cis.ohio-state.edu/hypertext/faq/usenet/wais-faq/\
             freeWAIS-sf/faq.html

comp.infosystems.wais FAQ
     Very old c.i.w FAQ. Some Pointers to clients.

     http://www.cis.ohio-state.edu/hypertext/faq/usenet/wais-faq/\
        getting-started/faq.html

This Manual

       * ftp://MIRROR-SITE/MIRROR-DIR/freeWAIS-sf-2.0/fwsf.texi.gz

       * ftp://MIRROR-SITE/MIRROR-DIR/freeWAIS-sf-2.0/fwsf.dvi.gz

       * ftp://MIRROR-SITE/MIRROR-DIR/freeWAIS-sf-2.0/fwsf.ps.gz

       * ftp://MIRROR-SITE/MIRROR-DIR/freeWAIS-sf-2.0/fwsf.info.gz

WWW95 Paper
     http://www.igd.fhg.de/www/www95/papers/47/fwsf/fwsf.html

Sources
=======

freeWAIS-sf-2.0
     ftp://MIRROR-SITE/MIRROR-DIR/freeWAIS-sf-2.0/freeWAIS-sf-2.0.tar.gz

waisperl
     For linking Perl with the freeWAIS-sf libraries. Used with SFgate. ftp://MIRROR-SITE/MIRROR-DIR/freeWAIS-sf-2.0/waisperl/Wais-2.0.tar.gz

SFgate
     The WWW Gateway for freeWAIS-sf.

     ftp://MIRROR-SITE/MIRROR-DIR/SFgate-4.0/SFgate-4.0.tar.gz

Example Format Files

     ftp://MIRROR-SITE/MIRROR-DIR/fmt-examples

Rainer Klute's Imake extension to CERN httpd 3.0

     ftp://ftp.germany.eu.net/pub/infosystems/www/cern/WWW-Imake.tar.gz

Building and Installation
*************************

Unpacking the Distribution
==========================

The file freeWAIS-sf-2.0.tar.gz is a tar archive compressed with GNU gzip. Unpack it with the following command:

     gunzip -cd freeWAIS-sf-2.0.tar.gz | tar xvf -

The directory structure is as follows:

                       |-FIELD-EXAMPLE
                       |
                       |-bin
                       |
                       |-ctype
                       |               |-CNIDR
                       |               |
                       |               |-SF
                       |-doc-----------|
                       |               |-original-TM-wais-|-manl
                       |
     |-freeWAIS-sf-2.0-|-ir
                       |
                       |-lib
                       |
                       |-regexp
                       |
                       |-ui
                       |
                       |-x

The directories contain

FIELD-EXAMPLE
     some test data and procedures, which check the system once you build it.

bin
     some shell scripts. Most of them for displaying non text documents after retrieval. These scripts are not maintained. They are included for convenience and drawn form freeWAIS-0.2.

ctype
     the code for the 8-bit character sets.

doc
     the documentation which accompanied freeWAIS-0.2 in the subdirectories CNIDR and original-TM-wais. In SF the documentation written for freeWAIS-sf can be found.

ir
     the main code parts for the indexer and the server.

lib
     some replacement of C functions which might be missing on some systems.

regexp
     the code for regular expression matching.

ui
     some standard clients

x
     the standard X11 client.

Running Configure
=================

The distribution comes with a Configure script generated by metaconfig which you should run next. It determines various system properties and asks you some questions.

Files generated
---------------

Configure generates some files:

config.h, confmagic.h
     This is where most of the option settings go. You may edit config.h and change some of them. This file is referenced via #include statements by most C files.

config.sh
     Here most setting for your directory layout and location of programs end up. This file is is sourced by all *.SH files when producing the corresponding files.

     You may change settings in config.sh and rerun Configure with the options -der.

Makefile
FIELD-EXAMPLE/Makefile
ctype/Makefile
doc/original-TM-wais/manl/Makefile
install
ir/Makefile
lib/Makefile
regexp/Makefile
ui/Makefile
bin/Makefile

     Makefiles for bulding and installing the distribution.

x/Imakefile

     Imakefile for building and installing the X11-Clients.

bin/catalog
bin/dictionary
bin/inverted_file
bin/makedb
bin/mkfmt
bin/ws

     Some scripts which depend on the path to perl.

Questions
---------

Run Configure by submitting the following to your shell:

     ./Configure

Configure will display a startup message and begin to check the required system properties.

Intermixed with  questions about variaus system properties there are some questions about the desired properies if the freeWAIS-sf system itself. Theese are explained in little more detail below.

Finally Configure prints the name of the files it generates and some short advice how to proceed.

Regular Expressions
...................

If your system has support for regular expression, the first question will be:

     ...
     Checking if your systems regexp implementation works ...
     Yes, it works
     Do you want to use your systems regexp.h? [n]

It is usually save to accept the default no and use the regular expressions included in the distribution. If that does not work (this maybe the case with IRIX) try your systems regular expressions by answering yes. In fact every answer other that no will be interpreted as yes.

Huge Headlines
..............

The next question refers to the length of the headline file. If you will not have large databases, answer no here. To estimate the size of your headline file, multiply the number of documents you want to index with the medium headline length. If this size fitz in three bytes, it is save to accept the default.

     Will you have HEADLINE files greater than 16 MB [n]

 -- config.h: bool MYREGEXP
     If the answer is different from no, a #define for MYREGEXP is added to config.h.

Proximity vs. String Search
...........................

Next decision selects one of two possible query extensions.

     This version of freeWAIS-sf supports new proximity operators by Tom
     Snee.  He also fixed the string search code.  You can now enable them
     at the cost of dropping the string search capability. There is
     currently no description of the proximity operators.  Have a look in
     ir/query_l.l to learn about them;

     Use proximity instead of string search? [n]

 -- config.h: bool PROXIMITY
     If the answer is different from no, a #define for PROXIMITY is added to config.h.

Character Set
.............

Configure now asks for the character set, you want to use. All characters for which your isalnum(c) returns true are assumed to be legal within words.

You can augment this character set by additional chars. If you want C++ to be a valid word, you must make + an allowed character. This mechanism is usually used to add country specific ISO characters to the default ASCII set.

     You can compile freeWAIS-sf with it's own ctype package. You should do
     this, if you want to use special (country specific) chars, which are
     not supported by your systems ctype.

     Use your systems ctype? [n]

 -- config.h: bool MYCTYPE
     If the answer is different from no, a #define for MYCTYPE is added to config.h. This also leads to the inclusion of the ctype directory in the building process. So a option -I../ctype is added to the compiler flags.

The following question is asked only if the default is accepted.

     I will now ask for your special letters.  If you do not want to give
     the now, edit config.h after this Configure run. Input your upper case
     letters in the same order than your lower case letters. toupper() and
     tolower() depend on this order. Input letters which are upper and
     lower case in both strings.
     what are your lower case letters ["a"o"u"s]?
     what are your upper case letters ["A"O"U"s]?

 -- config.h: string LCHARS

 -- config.h: string UCHARS
     The character strings are added as extension of LCHARS rsp. UCHARS. Note that the order of character matters, since the index position is used for the macros toupper(C) and tolower(C).

LOCAL_SEARCH
............

You can compile and link the clients with the capability to search index files directly. So you need not to install a server, for local searches. The clients will be greater, but faster with local searches.

You will have to run the test (see Tests) manually without the switch!

     Do you want to compile with -DLOCAL_SEARCH? [y]

 -- config.h: bool LOCAL_SEARCH
     Server code is linked to the clients when true.

Modified URL handling
.....................

You can modify the URL document type, to put the URL of the indexed document in the document id instead of the headline. If you use this modification you can customize the headline (e.g. with the -t fields option). Also it is not required to keep a copy of the documents for retrieval with the WAIS server. But currently only SFgate can handle this modified document-ids. Normal clients will not correctly interpret the document-ids and try to retrieve the document from a WAIS- instead of the corresponding HTTP server.  See Overview (The SFgate Manual).

     Do you want to use the modified URL handling? [y]

 -- config.h: bool URLDOCID
     The DocumentId is used to carry the URL to the client. The server will be unable to access the document. You can remove it after indexing.

Installation Prefix
...................

     Installation prefix to use? (~name ok) [/usr/local/ls6/wais]

Synonym Files
.............

I added a patch from Alberto Accomazzi for speeding up usage of synonym files. He writes about his patch:

     For those of you who have fairly large synonym files (> 10Kb) and are running the software on a machine that supports shared memory (your machine does) enabling this feature will speed up the waisserver response time by a significant factor.

     For those of you who do not have shared memory, I have rewritten the memory allocation part of synonym.c so that bigger memory chunks are allocated and used rather than allocating memory for each word and synonym, so the code should be a little faster for you too.

     Do you want to use shm cache? [n]

Registration
............

As distributed, freeWAIS-sf Release 2.0 will send an UDP packet to my workstation every time waisserver reindexes his info database, containing your (numeric) UID, your operating system, your compiler version and the freeWAIS-sf version.

This is just because I would like to get an idea of to which systems/compilers freeWAIS-sf has be ported and how many people use it, and keep on using it (rather than tried it once folks).  It will never become a licensing scheme or some crazy thing like that.  But, you can disable it by answering yes to the following question.  If you do that, please let me know, if you are running freeWAIS-sf on a system/compiler, which is not mentioned in the README.

     Disable the UDP packet sending? [n]
     Ok. Thank you for your trust.

Compiling the system and running the tests
==========================================

When you have generated the makefiles using Configure (see Configure), you can compile the system by just entering make all. This will also index a test database and run some tests. Here is the list of the make targets:

make
     Build the system and run the tests.

make install
     Install binaries and scripts.

make install.man
     Install the manual pages.

make clean
     Remove object files, libraries, backups, ...

make realclean
     Remove files generated by flex, bison, dvips, latex, waisindex.

If you need the x clients cd to the x subdirectory and enter xmkmf and make depend prior to make all and make install

Running the tests
-----------------

After successful compilation the make process indexes a test database and runs some queries. You will have to start the tests manually, if you did not chose the LOCAL_SEARCH option. See LOCAL_SEARCH.

     cd FIELD-EXAMPLE
     ../ir/waisserver -d . -p 9565
     make test
     ...
     kill WAISSERVER-PID

The results of the queries are compared to expected results contained in the distribution.  Depending on your choices during the configuration (see Configure) some of the tests will fail. These are mentioned below. The imake makefiles will avoid the test which are expected to fail.

Here is the list of tests:

PLAIN
     Probabilistic Indexing

BOOLEAN
     au=(pennekamp or fuhr) and processing

COMPLEX
     py==1995 and (ti=(Retrieval freeWAIS) or au=pfeifer)

FIELD
     au=pfeifer

DATE
     ed>930101

GERMAN
     au=großjohann

     This will only work if you made ß a legal chracter. See Character Set.

LITERAL
     'Enhanced Retrieval'

     This test will fail, if you chosed the proximity operators during configuration. See Proximity Search.

NUMERIC
     py==1995

PARTIAL
     Pfeif*

PROXIMITY
     (Searching w/2 Documents) or (Development pre/3 Linear)

     This test will succeed only, if you chosed the proximity operator during configuration. See Proximity Search.

If some of the tests supposed to work fail, have a look at the FIELD-EXAMPLE directory. For each test there is a pair of file: TESTNAME.is and TESTNAME.should. TESTNAME.should is what the query result was on my machine and TESTNAME.is was produced running on your system. Compare the files and decide, if you can live with the differences, e.g. if only the weighting differs, you should be able to use the system.

Installation
============

If the tests where successful, you can install the system using make install. See Make targets.

Installation Directories and Files
----------------------------------

This is how your installation directory should look like after installation. See Installation Prefix.

                           |-SFproxy
                           |-catalog
                           |-check-sources
                           |-dictionary
                           |-getaddrs
                           |-inverted_file
                           |-makedb
                           |-mkfmt
                           |-server_stats
                           |-stats.awk
                           |-stringtoany
                           |-swais
                           |-trunc
                           |-wais-gif-display
            |-bin----------|-wais-html-display
            |              |-wais-jfif-display
            |              |-wais-jpeg-display
            |              |-wais-pict-display
            |              |-wais-ppm-display
            |              |-wais-tiff-display
            |              |-waisindex
            |              |-waisping
            |              |-waisq
     prefix-|              |-waisretrieve
            |              |-waissearch
            |              |-waisserver
            |              |-ws
            |              |-xwais
            |              |-xwaisq
            |
            |              |-X11---------------|-app-defaults----|-Xwais
            |-lib----------|-XwaisHELP
            |              |-XwaisqHELP
            |              |
            |              |-emacs-------------|-lisp------------|-wais.el
            |
            |                                  |-catalog.1
            |                                  |-dictionary.1
            |                                  |-inverted_file.1
            |                                  |-makedb.1
            |                                  |-mkfmt.1
            |              |-man1--------------|-waisindex.1
            |-man----------|                   |-waisq.1
                           |                   |-waissearch.1
                           |                   |-waisserver.1
                           |                   |-xwais.1
                           |                   |-xwaisq.1
                           |
                           |-man3--------------|-ftw.3
                                               |-scandir.3

  * The .../bin directory must be added to your path.

  * The file .../lib/app-defaults/Xwais must be found by your X11 server. You can safely move it to some directory (e.g. /usr/local/X11/lib/X11/app-defaults) where your server already searches or add the .../lib/app-defaults/Xwais directory to the search path e.g. by adding .../lib/app-defaults/Xwais,%N to the environment variable XUSERFILESEARCHPATH.

  * The file .../lib/emacs/lisp/wais.el must be found by Emacs (see Overview (The GNU Emacs Manual)). You can safely move the file to your site lisp directory or add .../lib/emacs/lisp to your emacs load-path.

  * You should also add .../man to your MANPATH. Note that the manuals are out of date!

Installing the Server
---------------------

To install the server for being started by inetd proceed as follows. The description applies to SUNOS 4.1.3 but should be similar on other *NIX systems.

You do not have to install a server, if your clients are compiled and linked with the LOCAL_SEARCH (see LOCAL_SEARCH) option on. Without a server only clients which have access to the database files can do searches.

Add a line for the WAIS protocol to your /etc/services:

     wais           210/tcp

You can use any term instead of wais if you want. But z3950 is misleading (see Protocol). You can choose a port different from 210 if you want. Make sure, that your database descriptions contain the right port then (see Database Description).

To /etc/inetd.conf add a (one!) line:

     wais   stream  tcp     nowait  wais    \
             INSTALLDIR/bin/waisserver waisd.d -d DATABASEDIR  -e LOGFILE

Send the inet daemon a HUP signal:

     kill -HUP INETD-PID

Building Databases
******************

If you successfuly compiled and installed the system, you already have build your own database. This database will serve as an example throughout this section.

Let's first define some terms.

Document
     The entities you want to search for.

Source
     The files containing the documents you want to make searchable.

Index
     A set of files generated by waisindex which are used by the server to perform searches. See Index Files.

Database
     The combination of source and index.

Format
     The layout of the documents including the separation of documents in the files.

Field
     The Format description might associate regions in the documents with certain concept types. The sender of an email might be associated to the concept author. Searches might be constrained to such an concept type or field for short.

Type
     A type is attached to each document in the database. The possible types roughly correspond to mime content types. Possible types are TEXT, HTML, URL, ....

In general you have two options when indexing a new database: You can use a build-in format or use the -t fields option. The first alternative is not covered by this manual since the latter is more general and you can emulate the build-in formats with the field indexing.

waisindex Options
=================

The following is mostly drawn from the waisindex manual.

-d INDEX-FILENAME
     This is the base filename for the index files. Therefore if /usr/local/foo is specified, then the index files will be called /usr/local/foo.dct etc. The index should be stored on the local file system of the machine running waisindex.  It works over NFS, but it is much slower.

-a
     Append this index to an existing one.  Useful for incremental additions or updates.  This will only add onto an index, so that if a file has changed, it will get reindexed, but the old entries will not be purged. Therefore, to save space, it is a good idea to reindex the whole set of files periodically.

-r
     Recursively index subdirectories.

-mem
     How much main memory to use during indexing. This variable will have a large effect on how fast indexing is done.

-export
     This causes the resulting source description file to include the host-name and tcp-port for use by the clients.  Otherwise the file contains no connection information, and is expected to be used only for local searches.

-nocat
     Inhibits the creation of a catalog.  This is useful for databases with a large number of documents, as the catalog contains 3 lines per document.

-T TYPE
     Sets the type of the document to TYPE.

-t FORMAT
     This is the format of files that are handled by waisindex. To find out the list of currently known types, execute the waisindex command with no arguments and it will list them. Use -t fields to use the field indexing. The file DATABASE.fmt must contain the format description.

-stop
     This option refers to the list of otherwise valid words which will be excluded from indexing and therefor searching.

     If -stop is given the build-in stop list is not used. In any case the file DATABASE.stop might contain stopwords separated by newlines. If fact waisindex might add some words, if they occur to often.

FILENAME ...
     These are the files that will be indexed according to the arguments above.  To insure the files are registered in the filename list correctly, it is advised that these are full paths (beginning with a /).  If the database is used from a machine other than the machine on which the index is created, this should be a machineindependent path.

-stdin
     Read the names of the files to index from standard input instead  from the argument line.

-line
     Makes the proximity operators use the lines aus units instead of words.

Bulding a Format Description
============================

The trickiest part when building a new database is to write a format description --- the horrible DATABASE.fmt files.

An important thing to know when writing format descriptions is that waisindex parses sources files line by line. All of the nice regular expressions have to match a single line. So including multiple linefeeds in an expression will cause the matcher to fail.

Regular Expressions
-------------------

All regular expressions in the format description must be included between two /. Therefore a / in the regular expression must be escaped with a \. To match TCP/IP write /TCP\/IP/. This in consequence means, that a \ must be escaped too ;-). Other escape sequences interpreted by the parser are:

\n
     A newline character. See line by line matching.

\r
     The carriage return character.

\f
     The formfeed character

\X
     The characer C-X (control X) where X ranges from A to Z.

After escaping is resolved, the string is passed to a standard regular expression package (or your systems one see Regular Expressions).

Operator
     Meaning

X
     the character X

"X"
     an X, even if X is an operator

\X
     an X, even if X is an operator. Remember that the parser eats one \. So use /\\*/ to match a * in the text.

[XY]
     the character X or Y

[a-c]
     the characters a, b or c

[^X]
     any character but X

.
     any character but newline

^X
     an X at the beginning of a line

X$
     an X at the end of a line

X?
     an optional X

X*
     0,1,2, ... instances of X

X+
     1,2,3, ... instances of X

X|Y
     an X or a Y

(X)
     an X

X{M,N}
     M through N occurrences of X

Structure of the Format Descriptions
------------------------------------

A format description consist of three parts. The first part --- usually just one line --- describes how the files should be split into documents. The second part, called the Layout Section defines how a headline for each document should be computed.

The last and usually largest part defines which fields should be generated and how the indexing should be done.

Separation of Documents
.......................

A format description begins with a record-sep: directive.

     record-sep: REGEXP

Here REGEXP might be /\f/ to match a line containing a formfeed character. Note that the line matching the record-sep: regular expression is never indexed!

Layout Section
..............

The layout section must be embraced by layout: and end:. It can contain multiple headline: and one date: directive. Each headline: keyword defines a region in the document which should be copied to a section of the headline. The region, defined by the regular expressions which start and end it can occur any number of times in the document. The order of the directives defines the order of the sections in the headline.

     headline: START END WIDTH [SKIP]

The above line advises the indexer to copy the text between the matches for START and END, optionally skipping the text matched by SKIP to the next WIDTH characters of the headline.

The date: directives defines how a date to be associated with the document can be extraced. Note that this date is not usable for searches. It ist part of the headline and displayed to the user by some clients.

     date: START SCANFARG D-M-Y D-M-Y D-M-Y SKIP

The START and SKIP work like above. The SCANFARG parameter is passed to scanf to read in year, month, and day. The order of the arguments is indicated by the three D-M-Y where each might be year, month, and day. The month might be followed by string to indicate that the month will be given as a standard three-letter abreviation.

Here is complete example:

     layout:
     headline: /^PY: / /^[A-Z][A-Z]:/ 5 /^PY: */
     headline: /^AU: / /^[A-Z][A-Z]:/ 21 /^AU: */
     headline: /^TI: / /^[A-Z][A-Z]:/ 41 /^TI: */
     date:     /^ED: / /%d-%3s-%d/ day month string year /^ED: [^ ]/
     end:

Definition of Field and Index Types
...................................

The field specification part of the format description is made up of multiple region: ... end: groups.  Each is mapping a region of text to a set of query categories aka fields.

     region: START [SKIP]
        FIELDLIST OPTIONS INDEXSPECS
     end: END

The START, SKIP, and END expressions define the region of the document for which the index specification applies.

FIELDLIST is the list of fields each optionaly followed by a description string enlosed in double quotes ("). The description is entered in the database description (see Database Description).

Options include the directives numeric: SKIP WIDTH, date: D-M-Y D-M-Y D-M-Y, and stemming. The first option advises the indexer to allow numeric values only and makes sure that numeric atomic search expressions will work with the categories. The second advices the indexer to convert the date in the region to the yymmdd format prior to index it numerically.  The last option tells the indexer to stemm the words in the region before entering them in the index. When searching, the server also stemms search terms prior to the index lookup.

INDEXSPECS is a list of index types along with a keyword indicating if the region should be mapped to the designated categories (LOCAL), the default category (GLOBAL) or to both (BOTH). The default category is used when none is specified in the query. See Query Syntax.

Index types currently supported are TEXT, SOUNDEX and PHONIX.

Consider the following example:

     region: /^AU: /
             au "author names" SOUNDEX LOCAL TEXT BOTH
     end: /^[A-Z][A-Z]:/

To the indexer this means:

For all words starting with AU:  at the beginning of a line up to a line which starts with two capital letters followed by a colon and a blank, put the word in the default and the au category and its soundex code only in the au category.

Thus an author name can be found in the created database in the default category or the au category if the exact spelling is known. If the name is misspelled, it might be found using the query au=(soundex MISSPELLED-NAME). See Sample Format, Query Syntax.

The Index Files
===============

waisindex generates a number of index files. If called with waisindex -d test -t fields ... it uses:

test.fmt
     The format definition. See Format Description. This file is not necessary if the option -t fields is not used.

test.fde
     The optional format description. Plain text, which is added to the database description.

test.syn
     The optional synonym file contains multiple lines with synonym terms separated by spaces.

          wais freewais
          programming hacking

test.stop
     The optional stopword file contains words which should be ignored when indexing, each in one line.

The following files are generated or modified by waisindex:

test.src
     The database description. See Database Description.

test.fn
     The filename list. One entry for each file in the database. See Filename List.

test.hl
     The headline list. One entry for each document in the database. See Headline List.

test.doc
     Document table. One entry for each document in the database. Contains pointers to the filename list and the headline list. See Document Table.

test.cat
     The catalog file. One entry for each document in the database. A human readable combination of document table with headline list and filename list.This file may be very space consuming. You can avoid generating this file if you use waisindex with the -nocat option. See waisindex Options. See Catalog Files.

test.dct
     The global dictionary. One entry for each term in the default field. See Dictionary Files

test.inv
     The inverted file for the default field. For each term in the database, there is a list of postings giving the documents and positions in the documents where the term occurs. See Inverted Files.

test.stop
     waisindex might add words to the stopword file, if they occur to often and would break the index.

For each field in the format description a dictionary and a inverted file is generated. This applies only to waisindex run with the -t fields option.

test_field_NAME.dct

test_field_NAME.inv

The Database Description
------------------------

The database description (DATABASE.src) serves for two purposes:

  * The description is used by some clients to extract connection information. This might be hostname/ip-address and port or the location of the index files for local searches. See LOCAL_SEARCH.

  * A number of descriptions might be used to make up a database of descriptions also referenced to as directory of servers. Use waisindex -t server ... to build such a directory.

This is how a database descriptions looks like (when called without the -export option):

     (:source
        :version  3
        :database-name "/usr/local/ls6/src+data/src/freeWAIS-sf-2.0/FIELD-EXAMPLE/test"
        :cost 0.00
        :cost-unit :free
        :maintainer "pfeifer@buster"
        :keyword-list (
                       fuhr
                       pfeifer
                       )
        :fields (
                       (:field
                        :name "ck"
                        :description "Citation Code")
                       (:field
                        :name "py"
                        :description "Publication Year (numeric)")
                       (:field
                        :name "au"
                        :description "Author")
                       (:field
                        :name "ti"
                        :description "Title")
                       (:field
                        :name "jt"
                        :description "Journal Title")
                       (:field
                        :name "ed"
                        :description "Date of insertion (date)")
                )
        :description "Server created with freeWAIS-sf 2.0 PL 30 on Oct  6 18:09:24 1995 by pfeifer@buster
     The files of type fields used in the index were:
        /usr/local/ls6/src+data/src/freeWAIS-sf-2.0/FIELD-EXAMPLE/TEST
     Here goes the contents of the format description file (database.fde)
     "
     )

With the -export option the first part changes to:

     (:source
        :version  3
        :ip-address "129.217.20.190"
        :ip-name "buster"
        :tcp-port 210
        :database-name "test"
     ...

You can modify the database description, especially if you want to enter it in a directory of servers. Notice that waisindex might overwrite the description when re-indexing or adding to the database when the list of significant terms (keywords) changes. Therefore, use the format description file (DATABASE.FDE) to add your comments on the database.

The Filename List
-----------------

The filename list (DATABASE.fn) contains for each source file of the database its filename, modification time, and the type of the documents.

The files starts with four zero bytes. Then for each file follows it's name as zero terminated string, its modification time a four byte integer and the display type as zero terminated string.

Byte
     Contents

0-3
     0x00000000

4-52
     /usr/local/ls6/src/freeWAIS-sf/FIELD-EXAMPLE/TEST

53
     0x00

54-57
     0x2EEC461D (-> 13:46 Dec 12 1994 )

58-61
     TEXT

62
     0x00

The Headline List
-----------------

The headline list contains the headline; a null terminated string for each document of the database . The first 4 for bytes are zero.

Byte
     Contents

0-3
     0x00000000

4-73
      1990   Ait-Kaci, Hassan; Na  Implementing a Knowledge-Based Library I

74
     0x00

74-144
      1990   Shepard, Michael A.;  Transient Hypergraphs for Citation Netwo

145
     0x00

...

The length of the headlines is variable with an upper limit given by the macro MAX_HEADLINE_LEN in config.h.

 -- config.h: int MAX_HEADLINE_LEN
     Maximum length of a headline. It may cause difficulties when increasing the value above the default value of 300. Some of the problems are fixed; maybe not all ;-)

The Document Table
------------------

The document table contains one entry for each document in the database. Each entry contains a pointer to the filename list and the headline list. Also start and end of the document in the file is given. The other entries are  the number of terms and lines and the date extracted from the document (see Layout Section).

Byte
     Contents

0-1
     0x0000

2-4
     Three byte offset in the filename list.

5-7
     Three byte offset in the headline list. This entry may be four bytes if you selected the HUGE_HEADLINES option during configuration. See Huge Headlines.

8-11
     The first character of the document in the source file (four bytes).

12-15
     The last character of the document in the source file (four bytes).

16-19
     Number of term in the document (four bytes).

20-22
     Number of lines in the document (three bytes).

23-26
     Date extracted from the document (yymmdd: a long int year*10000+month*100+day).

27-29
     Next document: Three byte offset in the filename list.

...

The first 25 bytes entry might be zero filled as well. Therefor the entry for document N starts at N*25+2.

The Catalog Files
-----------------

Unless prohibited by the -nocat option (see waisindex Options), waisindex generates a human readable version of the document table (see Document Table) with the filenames and headline included from the filename list (see Filename List) and the headline list (see Headline List).

     Catalog for database: /usr/local/ls6/src/freeWAIS-sf/FIELD-EXAMPLE/test
     Date: Jun  4 13:32:46 1995
     276 total documents

     Document # 1
     Headline:  1990   Ait-Kaci, Hassan; Na  Implementing a Knowledge-Based Library I
     DocID: 0 244 /usr/local/ls6/src/freeWAIS-sf/FIELD-EXAMPLE/TEST

     Document # 2
     Headline:  1990   Shepard, Michael A.;  Transient Hypergraphs for Citation Netwo
     DocID: 244 437 /usr/local/ls6/src/freeWAIS-sf/FIELD-EXAMPLE/TEST

     ...

The Dictionary Files
--------------------

The dictionary file (DATABASE.dct, DATABASE_fieldNAME.dct contains entries for all valid terms in the database. An entry contains:

Bytes
     Contents

21
     The term as a zero terminated string. Therefor the length of the terms is 20 chars maximum.

4
     Pointer in the inverted file. See Inverted Files

4
     Should be the number of documents the term occurs in. Is the total number of occurances instead.

The entries are sorted by the terms and grouped in blocks of 1000 entries each. This main file is preceeded by a directory containing the first entries of the main blocks. The bytes 21-24 of this directory entries contain the pointer to the main blocks, the last four bytes are zero. Entries in the index block are sorted too so that binary search on the index can work. The first two bytes of the file give the number of main blocks used. E.g., if we have two blocks the layout is as follows:

Bytes
     Contents

0-1
     0x0000

2-3
     2

4-32
     Directory entry for the first block (21 Bytes: Term, 4 Bytes pointer, 4 Bytes zero)

33-61
     Directory entry for the second block

62-29061
     First block

29062-58062
     Second block

The Inverted Files
------------------

The inverted files are the heart of the index. They contain the references to the documents they occur in for each term. The first 4 bytes contain the number of all terms in the file. The rest of the file is made up of term entries. Note that the following is mainly debugger knowledge; Tungs documentation is not very good here. But a Perl script using this definition was able to parse my inverted files ;-).

Header1

     Bytes
          Contents

     1
          {, the dictionary flag.

     6
          0x000000000000. Probably used during building?

     2
          Length of header 1

     4
          Total number of occurances of the term. Multiple occurances in documents count!

     variable
          The term terminated by a newline. Variatio delectat.

Header 2
     This is where the entries in the dictionary point to. See Dictionary Files

     Bytes
          Contents

     1
          E the full flag, indicating a vaid entry.

     4
          Number of postings following this header.

     4
          Total size of the postings following this header.

Postings
     A list of postings, each referring to one document.

     Bytes
          Contents

     4
          The document id. This is the entry number in the document table. See Document Table.

     4
          The total size of the character position list including the first one(!). If you use the proximity operators (see Proximity Search) the  word position is stored instead. This also applies to the rest of the posting list.

     4
          A single precision float which gives the weight of the term with respect to the document. See Query Weighting.

     3
          The position of the first occurance of the term in the document.

     variable
          If there is more than one occurance of the term in the document it will be followed by a list of compressed integers. From each byte bits 6 to 0 are used. If 7 bit are not enough to store the number, bit 7 of the higher order bytes is set. So 0 to 127 is encoded as usual and 128 is encoded as 0x8100 129 as 0x8101 an so on.

Queries
*******

Formulating a Query
===================

As mentioned in Protocol, freeWAIS-sf uses the original WAIS Protocol. This protocol was designed just for transporting a free text query which means a list of search term separated by spaces. We decided deliberately to use this old protocol so that all clients out there could use our new features.

So we had to encode the new and richer query semantics in the query string. This means that the query now has to obey a certain syntax (see Query Syntax) and consequently a user might get a syntax error when submitting a query. So our goal was to make the syntax as easy as possible and especially leave simple free text queries valid.

In the query the categories to be searched should be selectable for each term. To leave the original queries valid (and to support casual users) we provided a default category, which is used if no category is specified in the query. Now we give an outline of the query language:

The atomic search expressions of the language are terms, term wild-cards and phrases (e.g. information, inform*, "information retrieval").

Stemming is handled transparently for the client. Terms searched in a stemmed category are searched using their word stem automatically. For a wildcard (only tail truncation is implemented), all matching words from the dictionary are used as search terms. Phrase search looks up the words in the string. At least one of them must be an index term. Then the server scans the documents containing this word for string matches for the complete phrase. This means that string search can only work if the server has access to the documents. For type URL this is not the case.

Additionally the prefix operators soundex and phonix are allowed for converting the following query term into its Soundex/Phonix code. This is for example very useful when searching in phonebooks if the exact spelling of a name is not known.  Arbitrary Boolean combination of these atomic expressions with the binary operators and, or and not (not means and not in Boolean logic and is therefore a binary operation, too. See the examples below.) are allowed. Parentheses can be used for grouping. For compatibility with the original syntax, or may be omitted.

For each expression, a semantic category (field) can be defined using the CATEGORY PRED operator, where PRED is = for text categories and one of =, <, > for numeric categories (== is also valid for backward compatibility to version 1.1).

Here are some examples:

information retrieval
     free text query

information or retrieval
     same as above

ti=information retrieval
     information must be in the title

ti=(information retrieval)
     one of them in title

ti=(information or retrieval)
     same as above

ti=(information and retrieval)
     both of them in title

ti=(information not retrieval)
     information in title and retrieval not in title

py==1990

py=1990
     numerically equal

py<1990
     numerically less

py>1990
     numerically greater

ed<19930101
     Date search. Format is yyyymmdd

au=(soundex salatan)
     soundex search, matches eg. Salton

ti=("information retrieval")
     phrase search

ti=(information system*)
     wild-card search

nuclear w/10 waste
     We have added a proximity search feature to freeWAIS-sf. With this feature, you could search for nuclear w/10 waste (like the Lexis/Nexis search syntax) to find all stories that have nuclear and waste within 10 words of each other.

nuclear pre/10 waste
     Proximity Searches

     If the order of the two words is important, then nuclear pre/10 waste will find all stories that have nuclear up to 10 words before waste. Proximity also works within fields; for instance, byline=(dan w/2 woods) will find every story with a byline that has dan within 2 words of woods. Note that you must use parentheses around the words you want to look for in the field.

atleast 20 clinton
     At Least Searches

     atleast 20 clinton finds every story that has at least 20 occurrences of clinton. atleast has to be all lower-case, and there cannot be any spaces between at, least, and the number -- at least 20 clinton will not work!

Query Weighting
===============

Let's for now disregard the boolean operators and assume that a query is simply a list of terms. Query weighting is done using the Vector Space model. Each term in the query is associated with a query term weight. Currently this weight is constantly 1. On the other side, the terms in each document get a document term weight. This weight is the product of a document specific weight and the inverse document frequency. The latter is defined as idf = log(N/N) where N is the number of documents in the database and N the number of documents the term occurs in.

The other part of the document weight is computed as follows: Let TF be the number of occurances of the term in document and MAXTF the maximum frequency of any term in the document. A preliminary weight is computed according to W = (0.5 * TF)/(1 + MAXTF). Then these weights are normalize by dividing them by the sum of the squares of all preliminary weights for terms in this document. So the document specific weights make up a vector of length 1. The final document term weight is yielded by multiplying this weight to the IDF.

For simple queries (no booleans) the weight of a document is computed by multiplying the query term weight to the query term weight for each term in the query and summing up the results. This is often referred to as the vector product (hence the name of the model) or scalar product.

Now let's get to the booleans. The or operator is just dropped. So information or retrieval yields exactly the same weight than information retrieval. To interpret the vector product in another way, you can say the or operator just sums up the weights of its arguments. For the and operator the weights of both arguments is computed and the final weight is just the minimum of these weights. Similar the binary not operator returns the minimum of the weight of the left argument and 1 - the weight of the right argument.

Remarks
-------

Numeric searches are currently handled the same way. This is clearly a bug. Why should a document from 1988 yield a higher weight with respect to a query <=1990 than one from 1987 just because there are more documents from 1987 in the database than from 1988?

I have no idea how strings searches are weighted.

Partial searches hopefully are weighted as if all matching terms where given.

The proximity operators might work like the booleans?

Linking freeWAIS-sf with other Systems
**************************************

WWW Clients
===========

Linking was tested with:

  * NCSA Mosaic 2.4

  * Tuebingen Univ Mosaic 2.4.2

WWW Server
==========

By Jean-Philippe Martin-Flatin <syj@ecmwf.int>

Linking was tested with:

  * CERN httpd 3.0

Direct WAIS access for CERN httpd 3.0 is easy to provide with freeWAIS-sf-2.0. The only thing you need to do is rename the WAIS libraries in CERN httpd Makefile's.

If you're a lucky guy and your system supports imake, you need to:

  * Retrieve Rainer Klute's Imake extension to CERN httpd 3.0. See Sources.

  * Replace WWW.cf in the top-level directory with the code in the Appendix (see freeWAIS-sf and CERN httpd)

  * Update the location of your freeWAIS-sf code in WWW.cf (variable WAISDIR)

If your system doesn't support imake, you need to update manually the WAIS libraries in the Makefile pertaining to your architecture and the top-level Makefile. And think about how easy life would be if only you had imake.

Gopher 2.1.1
============

By Steve Hsieh, originaly written for freeWAIS-sf-1.0

In gopher2_1_1/gopherd/Makefile:

replaced original SFWAISOBJ with:

linux

             SFWAISOBJ       = ../regexp/libregexp.a ../ir/libinv.a ../ir/libclient.a \
                               ../ir/libwais.a ../ir/liblocal.a ../ir/libsig.a \
                               ../ui/source.o ../lib/libftw.a

solaris & SunOS

             SFWAISOBJ       = ../ir/libinv.a ../ir/libclient.a ../ir/libwais.a \
                               ../ir/liblocal.a ../ir/libsig.a ../ui/source.o \
                               ../regexp/libregexp.a ../lib/libftw.a

In gopher2_1_1/gopherd/waisgopher.c: change

             MIN(

to

             MINIMUM(

Run Configure in freeWAIS-sf. See Configure.

For the following configure questions, required means that I had to use that value to get gopher to work with freewais-sf. doesn't matter means that the decision is up to you ....

     Do you want to use your systems regexp.h (no)?  no <-- required
     Will you have HEADLINE files greater than 16 MB (no)?   no <-- doesn't matter
     Use your systems ctype (no)?    yes <-- required
     Do you want to compile with -DLOCAL_SEARCH (yes)?       yes <-- required
     Do you want to use the modified URL handling (no)?      no <-- doesn't matter
     Where should the installation go (/usr/local/wais)? (specify your own path)
     Do you want to use shm cache (no)?      no <-- doesn't matter
     Disable the UDP packet sending (no)?    no <-- doesn't matter

Make freewais-sf ... create symbolic links to in gopher2_1_1 to the appropriate freewais-sf directories ...

In gopher2_1_1:

     ln -s ../freeWAIS-sf-1.0/ir
     ln -s ../freeWAIS-sf-1.0/ui
     ln -s ../freeWAIS-sf-1.0/lib
     ln -s ../freeWAIS-sf-1.0/regexp

Edit gopher2_1_1/Makefile.conf gopher2_1_1/conf.h as necessary for your system and site.

Make sure to uncomment -DFREEWAIS_SF in Makefile.conf !

make gopher ...

In the case of an index type not recognized error, test an index on an database that has been reindexed using the newly compiled waisindex in freeWAIS-sf-2.0/ir (as opposed to gopherindex) just to make sure that there really still is a problem...

Use waisindex -export ... It is important that the host and port information is compiled into the DATABASE.src file, even if the searches are local! I have my gopher, wais , and all the databases all on the same machine. It is also important that the waisserver is running.

Discussion of freeWAIS-sf
*************************

Newsgroup comp.infosystems.wais
===============================

Discussion of freeWAIS-sf mostly happens in comp.infosystems.wais. Please make sure that you have the term freeWAIS-sf somewhere in the subject line and the version you are referring to somewhere in the body. Please do not send mails to me. I will answer postings in c.i.w if they are of general interest - and I have the time. Do not reply to my postings by mail. Keep the readers of c.i.w informed by posting there.

freeWAIS-sf Mailing-list
========================

There is a mailing-list about freeWAIS-sf, which is intended for beta testers. Messages to wais@wsct.wsc.com will be forwarded to the subscribers. To join the mailing-list send a mail to majordomo@wsct.wsc.com:

     To: majordomo@wsct.wsc.com

     subscribe wais

To leave the mailing-list send the same mail with subscribe replaced by unsubscribe.

Bug Reports
===========

Send bug reports not referring the last version to the newsgroup (see Newsgroup comp.infosystems.wais).

If you are sure that you use the latest version, try to submit patches rather than bug reports. See Submitting Patches.

If you cannot solve the problem, send the bug report to the mailing-list. See freeWAIS-sf Mailing-list.

Include if possible:

  * Description of your hardware and operating system

  * Description how to reproduce the bug -- preferably with the included test database.

  * Debugger back-trace if the system crashes or of description of the misbehaving of the wais program if it just works not as it should.

  * The configuration options you chosed. See Configure.

Remember that clients are not the focus of freeWAIS-sf. So definitely we are only interested in patches for problems with the clients.

Submitting Patches
==================

If you submit patches, make sure that they apply to the latest version. If you have patches for old version, please verify that the problem persists with the latest version and try you path on it. Otherwise send the patch to the newsgroup (see Newsgroup comp.infosystems.wais).

Patches for the newest version should be send to the mailing-list. See freeWAIS-sf Mailing-list.

Try to send in context diffs like the ones generated by diff -c. Try to restrain from reformatting and indenting the source before generating the patch to restrict the differences to the real changes. Please also include diffs to the Changelog in each directory of the distribution. Add a description of your patch stating the problem, the solution, and the names of the functions you changed.

Sample Format Description
*************************

Sample document:

     CK: Frei/Schauble:91
     AU: Frei, H.P.; Schauble, P.
     TI: Determining the Effectiveness of Retrieval Algorithms.
     JT: Information processing & management.
     PP: 153
     ^L
     CK: Schauble:89
     AU: Schauble, Peter
     TI: Improving the effectiveness of retrieval systems by information
         structures.
     JT: Information processing & management.
     VO: 25
     PY: 1989
     NO: 4
     PP: 363

Sample format description:

     #                               -*- Mode: Sh -*-
     # test.fmt --
     # Author          : Ulrich Pfeifer
     # Created On      : Fri Oct  6 13:27:35 1995
     # Last Modified By: Ulrich Pfeifer
     # Last Modified On: Fri Nov  3 10:14:12 1995
     # Language        : FMW
     # Update Count    : 23
     # Status          : Unknown, Use with caution!
     #
     # (C) Copyright 1995, Universität Dortmund, all rights reserved.
     #
     # Remember: indexing and matching is done line by line. Each regular
     # expression can match one line maximum.

     # This regular expression should match the line separating documents.
     # The matching line is NOT indexed!  Use /\n\n/ if each file contains
     # only one record.

     record-sep: /\f/    # formfeed (alias for \L)

     # Now we specify how the headline is constructed.  The date is part of
     # the headline and not searchable.
     layout:

     # First me map the regions starting with /^PY: / up to (and excluding)
     # the next /^[A-Z][A-Z]:/ skipping everything from the beginning up after
     # the end of /^PY: */ to the first 4 characters of the headline.

       headline: /^PY: / /^[A-Z][A-Z]:/ 4 /^PY: */
       headline: /^AU: / /^[A-Z][A-Z]:/ 20 /AU: */
       headline: /^TI: / /^[A-Z][A-Z]:/ 40 /TI: */

     # The date is found on lines matching /^ED: /. The line is kipped up
     # to the end of /^ED: *./. The line is handled to scanf with format
     # string "%d.%d.%d". Day, month, and year are expected in this
     # order. Month may be a string containing the 3 letter abreviation for
     # the name of the month. In this case the keyword "string" must follow
     # "month"

       date: /^ED: / /%d.%d.%d/ day month year /^ED: *./

     # Now we define the first region. It starts at /^CK:/. everything up
     # after the end of /^CK: */ is skipped. Region ends at
     # /^[A-Z][A-Z]:/. The match for end is search BEHIND the match for the
     # begin.
     region: /^CK:/ /^CK: */

     # All words are transfered in the 'ck' index and the global
     # index. Description of 'ck' is overwritten by "Citation Code".

     ck "Citation Code" TEXT BOTH
     end: /^[A-Z][A-Z]:/

     # All words in this region will be numeric. They are merge in the 'py'
     # index which will be tagged as nummeric from here on (thats not how
     # it should be - I know). Make sure that you do not pollute 'ck' in
     # the following with non-numeric data.  The "numeric" keyword has it's
     # own skip expression /^PY: [^ ]/ which overwrites the region skip
     # expression. This way you can index the same region nonnumeric (This
     # not ...).

     region: /^PY: / /^PY: */
     py "Publication Year" numeric: /^PY: */ 4 TEXT LOCAL
     end: /^[A-Z][A-Z]:/

     # This region will be indexed plain for the global index and the 'au'
     # index. The 'au' index will also contain the soundex codes for the
     # words.

     region: /^AU: / /^AU: */
     au "Author" SOUNDEX LOCAL TEXT BOTH
     end: /^[A-Z][A-Z]:/

     # All words in theis reagion will be stemmend before insertion in the
     # global and the 'ti' index'

     region: /^TI: / /^TI: */
     ti "Title" stemming TEXT BOTH
     end: /^[A-Z][A-Z]:/

     # This region is indexed for multiple local and the global index.
     # Stemming applies for the 'ti' index and the global index.

     region: /^JT: / /^JT: */
     jt "Journal Title" ti stemming TEXT BOTH
     end: /^[A-Z][A-Z]:/

     # Here we make the data searchable. Format is merely the same as in
     # the layout section. There is no local skip expression. That wasnt a
     # good idea for numerics anyway. The date can be search with the
     # yymmdd format. See test DATE.

     region: /^ED: / /^ED: */
     ed "Date of insertion" <date> /%d.%d.%d/ day month year TEXT LOCAL
     end: /$/

     region: /^JT: / /^JT: */
     TEXT GLOBAL
     end: /^[A-Z][A-Z]:/

Query Syntax
************

     expression      : term
                     | expression or term
                     ;
     or              : %prec or
                     | or
                     ;
     and             : and
                     | not
                     ;
     term            : factor
                     | term and factor
                     ;
     factor          : unit
                     | unit pre/ unit
                     | unit w/ unit
                     | atleast/ unit
                     ;
     unit            : WORD-OR-LITERAL
                     | phonsound WORD-OR-LITERAL
                     | ( expression )
                     | WORD-OR-LITERAL  = ( s_expression )
                     | WORD-OR-LITERAL  =  WORD-OR-LITERAL
                     | WORD-OR-LITERAL  =  phonsound WORD-OR-LITERAL
                     | WORD-OR-LITERAL relop WORD-OR-LITERAL
                     ;
     relop           : = =
                     | <
                     | >
                     ;
     phonsound       : PHONIX
                     | SOUNDEX
                     ;
     s_expression    : s_term
                     | s_expression or s_term
                     ;
     s_term          : s_factor
                     | s_term and s_factor
                     ;
     s_factor        : s_unit
                     | s_unit pre/ s_unit
                     | s_unit w/ s_unit
                     | atleast/ s_unit
                     ;
     s_unit          : WORD-OR-LITERAL
                     | phonsound WORD-OR-LITERAL
                     | ( s_expression )
                     ;

Format Specification Syntax
***************************

     docspec         : parseformat | format
                     ;
     parseformat     : record layout
                     ;
     format          : record layout speclist
                     ;
     record          : record-sep: REGEXP
                     ;
     layout          :
                     | layout: layoutspecs end:
                     ;
     layoutspecs     : layoutspec
                     | layoutspec layoutspecs
                     ;
     layoutspec      : date: REGEXP REGEXP date date date regexp
                     | headline: REGEXP REGEXP INT regexp
                     ;
     speclist        : spec
                     | spec speclist
                     ;
     spec            : region: REGEXP regexp
                       fieldlist
                       options
                       indexspecs
                       end: REGEXP
                     ;
     options         :
                     | option options
                     ;
     option          : numeric: regexp INT
                     | date: date date date
                     | stemming
                     ;
     regexp          :
                     | REGEXP
                     ;
     indexspecs      :
                     | indexspec indexspecs
                     ;
     indexspec       : indextype dicts
                     ;
     indextype       : TEXT
                     | SOUNDEX
                     | PHONIX
                     ;
     dicts           : GLOBAL
                     | LOCAL
                     | BOTH
                     ;
     date            : day
                     | month monthspec
                     | year
                     ;
     monthspec       :
                     | string
                     ;
     fieldlist       :
                     | WORD descr fieldlist
                     ;
     descr           :
                     | " ANY STRING "
                     ;

freeWAIS-sf and CERN httpd
**************************

     /* Version numbers of library, daemon, and client: */
     #define LibwwwVersion 2.17
     #define DaemonVersion 3.0

     /* Where to install binaries? */
     BINDIR = /usr/local/infosystems/WWW/bin

     /* Do you want to compile the WAIS gateway? If yes, where is freeWAIS
        located?
     */
     #define FreeWAIS_sf YES
     #define FreeWAIS_02 NO
     #define FreeWAIS_03 NO
     #define FreeWais (FreeWAIS_02 || FreeWAIS_03)

     #if FreeWais
     /* WAISDIR = /usr/local/infosystems/freeWAIS-0.2 */
     WAISDIR = /usr/local/infosystems/freeWAIS-0.3
     WAISLIBDIR = $(WAISDIR)/bin
     WAISINCDIR = $(WAISDIR)/ir
     #endif

     #if FreeWAIS_sf
     WAISDIR = /usr/local/infosystems/freeWAIS-sf-1.0
     WAISLIBDIR = $(WAISDIR)/ir
     WAISINCDIR = $(WAISDIR)/ir
     #endif

     /*
      * If you don't want to use GCC for compilation, but the compiler defaulted in
      * Imake's configuration files, set UseGcc to NO. If you want to change the
      * native's compiler default options, set CCOPTIONS and/or CDEBUGFLAGS as
      * needed, preferrably depending on the architectur as show below.
      */

     #define UseGcc YES

     #if UseGcc   /* use GNU C compiler: */

              CC = gcc -fpcc-struct-return
       CCOPTIONS =

     #if defined (SunArchitecture) && !defined (i386Architecture) && !defined
     (SparcArchitecture)
       CCOPTIONS = -m68881
     #endif /* Sun-3 */

     #else   /* use native C compiler: */

     #if defined (HPArchitecture) && defined (hp9000s800)
     CCOPTIONS = -Aa -D_HPUX_SOURCE -Wl,+b $(USRLIBDIR) -Wl,+s
     #endif /* HPArchitecture */

     #if defined (XyzArchitecture) /* e. g. IBMArchitecture, SGIArchitecture etc. */
              CC = compiler-to-use
       CCOPTIONS = compiler-options
     #endif /* XyzArchitecture */

     #endif /* UseGcc */

     /* No need to modify anything below here */

      LIBWWWDIR = $(TOP)/Library/Implementation
         LIBWWW = -L$(LIBWWWDIR) -lwww
      DEPLIBWWW = $(LIBWWWDIR)/libwww.a

     #if FreeWais
        LIBWAIS = $(WAISLIBDIR)/client.a $(WAISLIBDIR)/wais.a
     DEPLIBWAIS = $(LIBWAIS)
     #endif

     #if FreeWaisSf
        LIBWAIS = $(WAISLIBDIR)/libclient.a $(WAISLIBDIR)/libwais.a
     DEPLIBWAIS = $(LIBWAIS)
     #endif

     /* Imake rule to create a generic name as a link */
     #ifndef InstallAsLink
     #define InstallAsLink(file,link,dir) @\
     install:: file                       @\
             set -x; cd dir; \            @\
             $(RM) link; \                @\
             $(LN) file link
     #endif