@(#) $Header: README,v 1.12 92/02/02 21:12:27 leres Exp $ (LBL)

    README for dexpire

This is the dexpire package. Dexpire is a replacement netnews expire
program. Most expire programs have you specify how long you want to
keep articles. Since netnews volume is constantly increasing, this
approach requires constant configuration file fiddling to avoid running
out of disk space. Also, when netnews volume slacks off (e.g. during
long holidays), traditional expires blindly free up too much space.

Dexpire dynamically determines how long to keep articles. You specify
the relative importance of hierarchies/newsgroups and how much free
space you'd like and it does the rest. This makes it easy to run on the
edge; articles are kept as long as possible without running out of disk
space.

Another advantage of dexpire is that it runs considerably faster than
traditional expires. It only looks the active file and spool
directories; run time is a function of the number articles deleted.
Since the history file rebuild is a seperate process, you have the
option of doing this less frequently (e.g. once a week).

This package has been tested with C News and INN (which is still in
beta at this time). If you use it with B News, I'd appreciate hearing
what changes are needed.

Hopefully, you should find the following files:

    README		- this file
    Makefile		- compilation rules
    dexplist		- sample control file
    dodexpire.cnews	- sample dexpire script (C News version)
    dodexpire.inn	- sample dexpire script (InterNetNews version)
    dohisttrim.cnews	- sample histtrim script (C News version)
    dohisttrim.inn	- sample histtrim script (InterNetNews version)
    dexpire.8		- manual entry
    histtrim.8		- manual entry
    runexp.8		- manual entry
    dexpire.c		- main program
    version.c		- release version number and date
    file.c		- active and dexplist parsers
    disk.c		- disk usage routines
    util.c		- random utility routines
    hash.c		- hash table utilities
    blocktest.c		- block size test program
    histtrim.c		- history file trimmer
    runexp.c		- process explicit expires
    dexpire.h		- configuration
    file.h		- forward declarations
    disk.h		- forward declarations
    util.h		- forward declarations
    hash.h		- forward declarations
    patchlevel.h	- patchlevel (just in case there are bugs)

Installation Instructions

We are a InterNetNews site and so these instructions are biased towards
this news system. This package is known to compile and nominally run
under SunOS 3.5 and SunOS 4.1 on Sun 3's and Sun 4's; under Ultrix,
thanks to Stan Barber. Dexpire uses statfs() to determine disk usage.
If you don't have statfs(), you'll have to write your own version of
disk.c. Since disk_usage() is only invoked once, it would be acceptable
to fork() and parse the output of /bin/df.

First test to make sure that dexpire's assumptions about the filesystem
block size are correct:

    make blocktest

This program checks to make sure that the st_blocks field of the stat
structure. Dexpire assumes that the units are in 512 byte blocks
(perhaps rounded up to the next even block because the filesystem
fragment size is 1024 bytes). If blocktest doesn't successfully build
and report "success," running dexpire might be dangerous. (I'm not
positive there are any Unix systems this test will fail on but it helps
me sleep better).

Next, configure dexpire.h. Although the location of the spool
directory, active and dexpire files can be changed with flags, it's
usually more convenient to have the builtins correct. It shouldn't be
necessary to change the DTIME or TOGO limits. MAX_FREE is a safety and
might need to be increased if you have a really, really small spool
partition.

Now configure the Makefile. If you don't have gcc, comment out the CC
line. It might also be necessary to change the target in the install
rule. The Makefile contains more detailed instructions.

Now configure a dexplist file. If you currently use the C News expire,
it's pretty easy to convert a explist file to the dexplist format;
basically, edit it down so that you only have the first 3 fields.
Otherwise, make a copy of the sample dexplist and start hacking.

Now compile and test. It's strongly recommended that you use -n until
you're sure everything works ok. Try something like:

    dexpire -vn -f 10000

Put the output on a file so you can examine it at your leisure. Make
sure the reported disk statistics match what /bin/df says. The first
pass should always find at least one article that could be deleted. The
end of the run should report a reasonable number of articles and the
correct number of bytes to be deleted.

The -d flag can useful in debugging a new dexplist file. It's a good
idea to check the output of:

    dexpire -vdn -f 10000

to make sure newsgroups end up in the classes you expect.

Include are sample scripts for both C News and InterNetNews. The are
good starting places as wrapper scripts for running dexpire and
histtrim. In particular, dohisttrim knows how to lock a the news system
and how to build the dbm/dbz history file.

If your new dexpire policy differs from your old expire setup, it isn't
unusual for there to be hundreds of consecutive unproductive passes.
But since dexpire caches article timestamps, these extra passes use
very little cpu time (and perform no I/O); and after the new policy
catches up, things will settle down to 10 to 20 passes.

Our news node is a Sun 3/60 with a Fujitsu 2266 running SunOS 3.5. Our
spool partition is about 800 Mbytes. It usually takes about 25 minutes
to do a daily dexpire and about 10 minutes once a week to trim and
rebuild our ~25 Mbyte history file.

Please send comments, suggestions, bug reports, etc to:

    Craig Leres
    leres@ee.lbl.gov (ucbvax!leres for uucp weenies)
    Lawrence Berkeley Laboratory
    One Cyclotron Road
    Mail Stop 46A-1123
    Berkeley, California 94720
