Author:     Johan Vromans 
Cmpny:      SQS 
Company:    Squirrel Software 
Dept.:      Public Domain 
Project:    Squirrel Mail Server 
Title:      Squirrel Mail Server
            Installation and Maintenance 
Version:    V3.01 
----------------------------------------

1.    INTRODUCTION 

1.1   The Mail Server 

      The mail server is a mail response program. This means that you 
      can send it an email message, and the program will read this 
      message, extracts commands from it, and execute these commands if 
      no errors were encountered. 

      The main purpose of the mail server is to handle requests for 
      files in archives. By sending a request for a file, the mail 
      server will look it up and send the requested file to the 
      originator of the request, either via email or via UUCP. 

      When files are transferred via email, binary files (e.g. 
      compressed archives) are encoded using one of several popular 
      encoding schemes. Big files are split into pieces to avoid mailer 
      limits and transmission failures due to communication problems. 

      The algoritms of the mail server are designed to satisfy user 
      requests as much as possible, without taking the risk that 
      unwanted information is sent. 

1.2   Mail Server Basics 

      The Squirrel Mail Server Software consists of a set of perl 
      programs, data files and a small wrapper program. The installation 
      procedure prepares these files, and stores them at a specific 
      location on disk, hereafter named LIBDIR. LIBDIR and the files it 
      contains should be owned by the so called 'mail server owner'. 

      The mail server itself is driven via an email alias entry in the 
      file '/usr/lib/aliases' (or whatever your system uses). 

      Default values are 'mserv' for the mail server owner, 
      'mail-server' for the mail alias, and '/usr/local/lib/mserv' for 
      LIBDIR. Al these (and more) can be configured for a particular 
      installation. 

2.    PRINCIPLES OF OPERATION 

      The main functions of the mail server are executed by the programs 
      'process' and 'dorequest'. 

      A third program 'listener' is usually used as a wrapper around 
      'process'. This little C-program disguises itself as the mail 
      server owner and executes 'process'. 

      The program 'process' reads an incoming email message, scans the 
      mail headers and examines the message body for mail server 
      commands. Valid server commands are stored internally until the 
      whole message has been read. If any errors were found, a short 
      help message is sent back to the sender of the message, and the 
      program quits. 

      If desired, messages originating from a pre-defined list of users 
      may be rejected without notice. The standard configuration defines 
      @black_list to a list of system accounts, e.g. 'root', 'news', 
      'daemon' etc. 

      If all commands appear valid, 'process' enters the second phase. 
      The commands are executed one by one. If files are requested 
      (which is normally the case) the files are looked up in the mail 
      server archives, using one or more search techniques discussed 
      below. If the file has been located, it is queued for delivery in 
      the mail server queue file. 

      If files are requested from remote FTP servers, a connection to 
      the server is made and the files are transferred to local storage. 
      The local copy of the file is then entered in the mail server 
      queue for delivery. 

      Finally, the program generates a report and sends it to the sender 
      of the message. This report contains (a.o.) details about the FTP 
      activities, and a list of files that are queued for delivery. 

      At this point 'process' is complete and exits. 

      Some other time, program 'dorequest' will be run, either 
      automatically from 'process' or via cron. This program handles the 
      actual delivery of files. It is also responsible for packing 
      directories and removing temporary files. 

      For delivery via electronic mail, the requested file is (usually) 
      encoded to protect binary information from text-based mail 
      programs. The encoded file is chopped into parts (chunks) of a 
      specific maximum size, to not exceed mailer limits. The parts are 
      then passed to the mail program for delivery. If only some parts 
      need to be delivered, only these parts are passed to the mail 
      program. 

      For delivery via UUCP there is no need to encode, so the actual 
      input file is passed, in pieces if necessary, to the delivery 
      program 'uucp'. 

      An entry is made in the mail server logfile for each (partial) 
      delivery. 

2.1   Format of the queue file 

      Each queue entry takes one record from the queue file, with tab 
      separated items as follows: 

         type           'M' or 'm' for email delivery,
                        'U' or 'u' for UUCP transfer,
                        'MP' for packed email,
                        'UP' for packed UUCP.
                        If the 'm' or 'u' is used instead of 'M' or
                        'U', the file is removed after being handled.
         recipient      the reply address used
         destination    email address (for email), or host!path for UUCP
                        transfers.
         notify         UUCP notification user
         request        the request
         file           the real name of the real file
         encoding       Encoding:
                        'B' = btoa, 'U' = uuencode, 'X' = xxencode,
                        'D' = Dumas' uue
                        If a 'Z' is appended, the file will be compressed
                        before being transferred.
         limit          max size (in Kb) per chunk
         parts          comma-separated parts list.

2.2   Format of the logfile 

      If you select logging, all transfers that are sent are logged in 
      the logfile. Each record has a number of (space separated) fields 
      as follows: 

         date           e.g. 920501
         time           e.g. 15:22
         type           'M' for email, 'U' for UUCP transfer
         destination    email address, or host!notify for UUCP transfers
         request        the name of the requested file
         Xnn/mm         X = encoding;
                        nn/mm = part nn of mm parts
         size           the size of the transfer

      Errors are logged with type 'F'. The remainder of the record 
      contains the text of the error message. 

      If an error is detected due to a user request, the queue entry for 
      this request is entered in the logfile with type 'Q'. If the 
      failure is temporary, the queue entry can be extracted from the 
      logfile and added to the queue (or better: queue~) file. 

      The support programs programs 'report.pl' and 'do_report.pl' can 
      be used to generate reports from the logfile. 

3.    IMPLEMENTATION DETAILS 

      Program 'process' uses the package 'rfc822' to read the incoming 
      mail message. This package supports reading header and body of the 
      message, and analyzing RFC822 compliant message headers. 

      One of the functions of the package is to extract the send 
      information from the 'From', 'Sender' and 'Reply-To' headers. 

      The commands are parsed by subroutine 'command_loop'. The command 
      handling is table driven, %cmd_tbl contains, for each of the 
      command verbs, the name of the subroutine to be called to parse 
      the command. 

      Upon completion subroutine 'index_loop' (required from file 
      pr_doindex.pl) is called to handle index requests stored 
      internally in @indexq. 

      Then subroutine 'search_loop' is called to handle search requests 
      stored in @searchq. 

      If needed, subroutine 'do_work' (required from file pr_dowork.pl) 
      is called to handle the commands stored internally in @workq. This 
      will probably result in transfer requests being stored in @queueq. 

      Processing @workq is table driven, just like the command loop. 
      Therefore is it very easy to add your own commands (and execution 
      code) to the mail server program without having to modify the 
      source code. See chapter "Advanced Techniques" for more 
      information. 

      If @queueq contains transfer requests, these requests are reported 
      and stored in the queue file for processing by program 
      'dorequest'. 

      Finally, a help message is generated on demand, or if syntactic 
      errors were detected by 'command_loop'. 

3.1   Archive file lookup 

      One of the advantages of the Squirrel Mail Server is the 
      possibility to spread the archives over a number of file trees. 
      Another advantage is the ability to search for files in the 
      archives, using file- and directory searches and index lookup. 
      While searching, a number of known extensions are appended to the 
      filename so that a request for 'foo' might result in finding 
      'foo.Z' or 'foo.tar'. 

      The general principle of searching is: 

      -- Use file lookup;
      -- if not found; use directory search;
      -- if still not found: search the index;
      -- unsuccessfull, or multiple possibilities were found,
         try automatic packing;
      -- if still unsuccessfull, try automatic compression.

      When a SEARCH command is issued, all alternatives are tried and a 
      list of possibilities is returned. 

3.1.1 File search strategy 

      This strategy is rather straightforward: for each of the archive 
      directories, try to locate the requested file by appending the 
      known extensions. E.g. when looking for 'file': 

            dir1/file
            dir1/file.tar
            dir1/file.tar.Z
            ...
            dir2/file
            dir2/file.tar
            ... etc ...

      If an exact match is found, the search within the directory is 
      terminated. This way it is possible to locate 'foo' even if 
      'foo.tar' also exists. 

3.1.2 Directory search strategy 

      The directory search extends the file search. The code is located 
      in file 'pr_dsearch.pl'. 

      It reads the directories that are specified in @libdirs, and 
      searches them for entries that correspond to certain patterns. In 
      general, it looks for filenames that start with the search 
      argument, optionally followed by something that can be interpreted 
      as a version number. If such a filename is found, and it 
      corresponds to a directory, the search recurses to this directory. 
      If not, the known extensions are tried. 

      E.g. when looking for 'file': 

            dir1/file-2.3.tar.Z              <-- found
            dir1/file/                       <-- recurse
            dir1/file/file-2.3.tar.Z         <-- found
            dir1/file-2.3/                   <-- recurse
            dir1/file-2.3/file.tar.Z         <-- found
            dir1/file-2.3/file-2.3.tar.Z     <-- found
            ...

      This process is repeated for every directory in @libdirs. 

3.1.3 Index search strategy 

      The final (and most powerful) search strategy uses a list of 
      filenames, and searches for every occurrence of the search 
      argument. The code is located in file 'pr_isearch.pl'. 

      The index strategy is based on the GNU 'locate' program. This 
      program provides an extreme fast searching method in a compressed 
      list of files. 

      When searching the index, the search argument is turned into a 
      regular expression pattern and passed to the 'locate' program. The 
      output of this program is further analyzed to find filenames using 
      the same method as described above (version numbers and known 
      extensions). 

      For efficiency, the 'locate' database contains the file date and 
      size, so no other lookups are needed. 

      It is possible to have an index file per archive directory 
      (default situation), or one single index file that contains 
      information for all archive directories. 

      The index information is maintained by the program 'makeindex'. 
      This program uses GNU 'find' to traverse the archive directories, 
      and GNU 'locate' tools to create the index files. 

3.1.4 Automatic Packing 

      This section only applies if $auto_packing and $packing_limit have 
      been set in the configuration file. 

      If the user requests for 'file.XXX', where XXX is a known packing 
      extension ('zip', 'zoo', 'tar' or 'tar.Z'), and this file does not 
      exist, automatic packing is tried. 

      The packing extension is removed from the filename. If the 
      resultant name points to a directory it is handled as if the 
      request were preceeded with a PACK command. See also the section 
      on Packing, below. 

      No search strategies will be used when locating this directory. 

3.1.5 Automatic Compression 

      This section only applies if $auto_compress and $compress have 
      been set in the configuration file. 

      If the user requests for a file 'foo.Z', which does not exist, but 
      a file 'foo' exists in the archives, this file is queued for 
      delivery. Before transfer, it will be compressed. 

      The strategies file search, directory search and index search will 
      be used when locating this file. Normally, no known extensions are 
      tried. By setting $auto_compress to a value greater than 1 in the 
      config file, known extensions will be tried, so that a request for 
      'file.Z' may result in 'file.shar.Z' (or 'file.zoo.Z'). 

3.2   FTP file lookup 

      The Squirrel Mail Server uses a cached FTP lookup mechanism. The 
      code is located in file 'pr_ftp.pl'. 

      When a request for a file to be retrieved via FTP is handled, the 
      mail server performs the following steps. 

      1. A connection is made to the FTP host. 

      2. A directory request (LIST command) is used to retrieve 
         information about the file that is being requested. 

      3. The file is looked up in the FTP cache. If the file exists in 
         the cache, the file modification date and size are compared to 
         the FTP information from the FTP host. 

      4. If the file is not in the cache, or if the info does not match, 
         the file is retrieved from the FTP host and placed in the 
         cache. 

      5. The file in the cache is queued for delivery. 

      This caching strategy is based on the assumption that, when a 
      piece of interesting software is made available via anonymous FTP 
      and announced on the net, several requests for this file will be 
      made within a short period of time. 

      Special precautions are taken to prevent problems when the cache 
      does not exists, or is write-protected, etc. In this case, the 
      file is retrieved to temporary storage and transferred from there. 

      The filenames for the cache are generated by reversing the 
      dot-separated parts of the FTP host name, and prepending these (as 
      directories) to the file name, e.g. 

            prep.ai.mit.edu:/pub/gnu/emacs-18.59.tar.Z

      is stored in the cache as 

            edu/mit/ai/prep/gnu/emacs-18.59.tar.Z

      A leading directory "pub" is removed from the resultant filename. 
      The host info is transformed to all lowercase; the filename is 
      not. 

      NOTE:  VMS style filenames are recognized and transformed. The
             resultant filename is lowcased.
             VMS directory info is not handled yet.

3.3   Directory listings 

      Directory listings are obtained from the server using a local 
      "ls -l" command or an FTP "LIST" command, the return info is 
      stored in a temporary file. If the size of this file does not 
      exceed a specific limit ($fb_limit) it is included in the output. 
      Otherwise it is queued for delivery. 

3.4   Archie commands 

      Archie commands are handled by executing the 'archie' program. The 
      output is stored in a temporary file. If the size of this file 
      does not exceed a specific limit ($fb_limit) it is included in the 
      output. Otherwise it is queued for delivery. 

3.5   Packing 

      With packing, it is possible to gather the contents of a directory 
      into a single file, which is then queued for transfer. Before 
      doing so, the size of the resultant file is calculated. If the 
      size exceeds a specific limit ($packing_limit) the request is 
      rejected, to prevent accidental (or not accidental) retrieval of 
      too much information. 

      The name of the directory is stored in the queue file, the actual 
      packing and delivery is handled by the 'dorequest' program. 

3.6   The delivery process 

      Upon completion of 'process' all files to be transferred are 
      stored somewhere on disk, and their names and destinations are in 
      the queue file. 

      The delivery is handled by program 'dorequest'. It can be run 
      automatically from 'process', from 'cron' or manually. 

      Upon invokation, it ensures that there is only one instance of the 
      program running. Then it locks the mail server queue, reads it 
      into internal storage, writes it out to a special backup version 
      on disk, empties the queue on disk and releases it. The queue file 
      is not simply removed, since other instances of 'process' may 
      already be waiting to store new entries in the queue file. 
      (Actually, 'dorequest' picks up an existing backup on disk first, 
      since that file results from a previous run of 'dorequest' that 
      was not properly finished, most likely due to a system crash.) The 
      backup file is hereafter referred to with 'queue~', since its name 
      is dereived from the name of the queue file by appending a tilde. 

      Next, the program starts to handle the entries that were in the 
      queue. First the internal copy of the queue is written to the 
      queue~ file. Then an entry is taken from the internal queue, to be 
      handled. This way, if the system crashes during a 'dorequest' run, 
      no requests will get lost. 

      If the request needs packing, the contents of the directory are 
      stored into a single file, which is then queued for transfer. 

      The following packing methods are supported by the mail server, 
      although they need not be present in each implementation. 

      *  (Compressed) tar 

         The contents of the directory are gathered into a tar-file, 
         which is then compressed. If the public domain program 'pdtar' 
         is available, it is ised to handle the packing and compressing. 
         Otherwise the output of 'tar' is piped to 'compress'. 

      *  Zoo 

         The popular 'zoo' program is used, in combination with 'find', 
         to gather the information. 

      *  Zip 

         The INFO-ZIP standard 'zip' program is used to handle the 
         request. 

      If the requested file must be compressed, this is handled now. The 
      file is compressed into a temporary file which is further 
      processed. If the file to be transferred was a temporary file, it 
      is removed. The (freshly created) temporary file is marked for 
      deletion so it will be removed upon completion of this request. 

      For delivery via electronic mail, the requested file is (usually) 
      encoded to protect binary information from text-based mail 
      programs. The encoded file is chopped into parts (chunks) of a 
      specific maximum size, to not exceed mailer limits. The parts are 
      then passed to the mail program for delivery. If only some parts 
      need to be delivered, only these parts are passed to the mail 
      program. To reduce system load, the 'dorequest' program sleeps a 
      bit between parts so the mail delivery can take place. 

      For delivery via UUCP there is no need to encode, so the actual 
      input file is passed, in pieces if necessary, to the delivery 
      program 'uucp'. 

      An entry is made in the mail server logfile for each (partial) 
      delivery. 

      Finally, if the file to be transferred was a temporary file, it is 
      removed and processing continues with the next entry in the queue. 

      Upon completion, the backup of the queue file (queue~) on disk 
      will be removed. 

4.    SUPPORT PROGRAMS 

4.1   Generating the index files 

      The program 'makeindex', as its name suggests, maintains the index 
      information. It is based on the 'updatedb' program that comes with 
      GNU find. 

      The program should be run from cron, as often as needed to keep 
      the index file synchronized with the actual contents of the 
      archives. Usually once per day (night) is sufficient. 

      Sometimes it is not appropriate to include all information in the 
      index file, e.g. a specific sub-tree might contain your local news 
      spool which is not to be retrieved via the mail server. You can 
      set @libprunes in the configuration file to a regular expression 
      pattern to exclude certain directories from being traversed. Note 
      that this pattern is implicitly anchored, i.e. it behaves as if it 
      were placed between '^' and '$'. @libprunes, if used, should 
      contain a pattern for each of the directories in @libdirs, in the 
      same order. 

4.2   Analyzing the log file 

      The support programs programs 'report' and 'do_report' can be used 
      to generate reports from the logfile. 

      'do_report' is a wrapper around 'report', and will usually be the 
      only program used. 

      When run, 'do_report' gathers logfile information about transfers 
      and errors. Both reports are mailed to the mail server owner by 
      default. You can specify other recipients on the command line. 

      You can specify a cutoff date and time for error reporting by 
      supplying the name of a file that is timestamped after the 
      previous error report. The 'do_report' program normally updates 
      the timestamp of this file, so errors are reported only once. 

      A typical usage report looks like: 

            Mail Server Report for July 1992 -- by Package                           Page 1
            
                                                                     1111111111222222222233
            Package                              Type Total 1234567890123456789012345678901
            -------------------------------------------------------------------------------
            /usr/local/lib/mserv/pub/README         M     1                   S
            /usr/local/lib/mserv/pub/mserv.tar.Z    M     2                   S
            TOTAL                                         4                   S

      'Type' will be 'M' for mail transfers, and 'U' for UUCP transfers. 
      'Total' denoted the number of times a file has been requested. The 
      rest of the information shows on what days requests have been 
      issued. In the above example, all requests were issued on July 19 
      (a Sunday). 

      Likewise, a list will be generated containing the users that have 
      requested files. 

      Program 'do_report' can also be used to cleanup the logfile. If 
      this function is selected, the logfile is renamed to a '.o' 
      extension (e.g. 'logfile.o'), and appended to a file with a '.cum' 
      extension (e.g. 'logfile.cum'). It is possible to compress the 
      accumulating logfile, in this case the program will uncompress is 
      before appending the new information, and compress it afterwards. 

      Command line options for 'do_report' are: 

                -config XX   use alternate config file
                -usage       generate usage report
                -ftp         show files in FTP cache
                -full        generate report for usage, errors and ftp
                -ftpclean    cleanup old files in FTP cache (implies -ftp)
                -ftpkeep NN  number of days a file is to be kept in the FTP cache
                -since FILE  only error messages newer than FILE
                             (FILE date will be updated upon successful completion)
                -noupdate    do not update FILE date
                -collect     collect and cleanup logfile data
                -help        this message
                -trace       show commands
                -ident       print identification
            
            Default action is to generate a usage report, and to mail it to the
            recipients.

4.3   The FTP cache 

      Program 'do_report' can also be used to report the contents of the 
      FTP cache, and to exipre it. 

      A typical report, generated with "do_report -ftp", will look like: 

            Files in FTP cache /usr/local/lib/mserv/pub
            
              Timestamp     Age* Size  Filename (* means: file has been removed)
            --------------  ---  ----  ------------------------------------
            92/12/05 02:24   5   142K  com/foo/ftp/crack-4.1.tar.Z
            92/12/01 12:34   9*    2K  com/foo/ftp/pasta/INDEX

      'Timestamp' indicates the last modification time of the file as 
      dereived from the FTP server. 'Age' is the last access date of the 
      file in the cache, in days. The cache expiry time is set in the 
      configuration file, but can be overridden with a command line 
      option. By using the last access date for expiry, the file will be 
      considered 'new' each time it is accessed. An '*' after the file 
      age means that this file has exceeded the expiry time and has been 
      removed from the cache. 

5.    INSTALLING THE SQUIRREL MAIL SERVER SOFTWARE 

5.1   Preparation 

      As distributed, perl is expected to reside in /usr/local/bin. You 
      can change this in the Makefile if your perl resides somewhere 
      else. 

      The mail server needs a user id to own the programs, and a place 
      where the programs and files can be stored. Pre-configured values 
      are "mserv" and "/usr/local/lib/mserv" respectively. We'll call 
      the library LIBDIR in this document. LIBDIR could be the HOME 
      directory of the mail server owner, but it need not. If you prefer 
      a different value for LIBDIR, change the appropriate line in the 
      Makefile. The name of the mail server owner can be changed in step 
      2, as described below. 

      Since all messages sent by the mail server are sent under the 
      server owner uid, its full name (the GCOS or comment field in 
      /etc/passwd) should be descriptive. 

      The mail server owner does not need any privileges, except for 
      read access to the archives, and read/write access to LIBDIR. 

      Do not use the root account for server owner! 

5.2   Configure and build 

      Tailor 'ms_config.pl' to reflect your specific needs. It has 
      extensive descriptions for configuration variables and values. Do 
      not forget to tailor the Makefile also, if needed. 

      It may be a good habit to generate differences using the 'diff' 
      program so you can apply your modifications to a new version of 
      the mail server software easily. 

      Issue "make" to prepare the programs. 

      If you select the index file feature, you need GNU find / locate. 
      As of GNU find version 3.6, you can use the locate program 'as 
      is'. For version 3.5, change GNUFIND in the Makefile to designate 
      where your GNU find sourcetree is, and issue "make ixlookup". This 
      will build a customized version of GNU locate, called 'ixlookup'. 

      Issue "make listener" to generate and compile the listener 
      program. This command uses ms_config.pl (from the current 
      directory) to determine how to build the listener program. 

5.3   Verify the configuration 

      You can now verify the configuration with the 'chkconfig' program. 

      You may need to run the perl tool 'h2ph' to supply system include 
      files needed for locking and socket IO. Pay attention to the lines 
      that got a "Not found" remark. Since the software has not been 
      installed yet, some of these remarks must be considered normal. 

            % perl chkconfig.pl -current
            Squirrel Mail Server Software V3.01 [chkconfig 1.30]
            MSERVLIB = (not set, but that's OK)
            Program library: /usr/local/lib/mserv/src [OK]
            
            Server owner: mserv, uid = 50, gid = 8, "Squirrel Mail Server"
            Bcc user: mserv, uid = 50, gid = 8, "Squirrel Mail Server"
            
            The 'listener' program will use the setruid system call
              It will change identity to compiled-in uid 50
              (setenv will be used to set USER, LOGNAME and HOME)
            The 'listener' program is not installed correctly!
              It should be installed setuid mserv
            
            Replies will be sent using "/usr/lib/sendmail" [OK]
            Preset mail headers:
              From: PRONTO Mail Server <bit-bucket@mh.nl>
              X-Server: Squirrel Mail Server Software V3.01 [chkconfig 1.30]
              X-Info: Send mail to <postmaster@mh.nl>
            
            Mail messages from the following users will be rejected:
              root uucp mailer MAILER-DAEMON news daemon demon deliver sendmail
            
            Transfer strategies:
              email: "/usr/lib/sendmail -odq" [OK]
                     wait 30 seconds between chunks
                     limits: 64K (default), 10K (min), 1024K (max)
              uucp : "/usr/bin/uucp -ga" [OK]
                     check host names using "/usr/bin/uuname" [OK]
                     limits: 256K (default), 10K (min), 2048K (max)
                     uucp transfer is preferred, if possible
            
            Search strategies: File Directory Index
            
            Archives:
              /usr/local/src [OK]
              /beethoven/arch [OK]
              /albinoni/arch [OK]
              /usr/local/lib/mserv/src/pub [*** Not found ***]
            
            Indexfiles:
              /usr/local/src/ix.codes [Not found, but that's OK]
              /beethoven/arch/ix.codes [Not found, but that's OK]
              /albinoni/arch/ix.codes [Not found, but that's OK]
              /usr/local/lib/mserv/src/pub/ix.codes [Not found, but that's OK]
            Limit per index request: 200 lines.
            
            Index tools:
              gfind      /usr/local/bin/gfind [OK]
              ixlookup   /usr/local/bin/locate [OK]
              locatelib  /usr/local/lib/locate [OK]
            
            Server files:
              queue      /usr/local/lib/mserv/src/queue [OK]
              logfile    /usr/local/lib/mserv/src/logfile [Not found, but that's OK]
              lockfile   /usr/local/lib/mserv/src/lockfile [OK]
              notes      /usr/local/lib/mserv/src/mserv.notes [OK]
              hints      /usr/local/lib/mserv/src/mserv.hints [OK]
            
            Locking with fcntl(2).
            Use the 'testlock' program to verify the locking! (See INSTALL)
            
            Default encoding is B (btoa)
            Encoders:
              btoa       /usr/local/bin/btoa [OK]
              uuencode   /usr/bin/uuencode [OK]
              uue        /usr/local/bin/uue [OK]
              xxencode   /usr/local/bin/xxencode [Not found, but that's OK]
            
            Tools:
              dir: "/bin/ls -lL" [OK]
              archie: "/usr/local/bin/archie" [OK]
              extensions: /usr/local/lib/mserv/src/userdefs.pl [OK]
              feedback limit = 8K
            
            Support for FTP is included.
              Limited to uucp only.
              Cache dir = /usr/local/lib/mserv/src/ftp [*** Not found ***]
              Files are kept for 8 days in the cache.
            
            Support for packing is included.
              Packing limit = 4100 blocks.
              Dusk usage obtained using "/bin/du" [OK]
              File list obtained using "/usr/local/bin/gfind" [OK]
              Methods: tar [OK] zip [OK] zoo [OK]
              Compress/Tar using "/usr/local/bin/pdtar" [OK]
              Automatic packing of directories is allowed.
            
            Working storage: /usr/tmp [OK]
            
            The queue will automatically be run upon completion of process.
            The mail server will be niced with 10.

5.4   Installation 

      Now "su mserv" and execute "make install". 

      This will install all programs and files in LIBDIR. It will not 
      install programs ixlookup and listener, since they need special 
      treatment. 

      If you select the index file feature, issue "make 
      install-ixlookup" if you need the customized version of GNU locate 
      3.5. Note that whoever creates the index files, must have write 
      access to the location the index files are stored. 

      If you configured the listener to use the setruid/setrgid system 
      calls, you need to "su mserv" before executing "make 
      install-listener". 

      If you do not have setruid/setrgid, the program must be installed 
      setuid root. In this case, you have to "su root" before doing 
      "make install-listener". Look at the source of the listener.c 
      program to verify it will not harm your system security. 

      Cd to the LIBDIR directory, and run 'chkconfig' again. Make sure 
      that all settings have the expected values. 

5.5   Testing interactively 

      Run LIBDIR/listener als follows: 

            % ~mserv/listener -noqueue

      Is should display a friendly banner: 

            Mail Server (Squirrel Mail Server Software V3.01) ready.
            Local time is Thu Dec 24 20:33:06 1992.
            Enter HELP for a list of commands.
            
            Command>

      Type "mail foo", followed by a RETURN: 

            Command> mail foo
            Command: mail foo
            => Transfer via email to "foo"

      Try "send HELP", followed by a RETURN: 

            Command> send HELP
            Command: send HELP
            => Send: HELP
            Request results:
            
              Request                   Size  Enc  Limit  Status
              -----------------------  -----  ---  -----  ------
              HELP                       19K   A     64K  Tested OK
            
            Encoding A means: not encoded (plain file).
            
            The requests with status "Queued" will be sent as soon as
            the load of the server system permits, usually within 24 hours.

      At this point, the basic functions of the mail server are working. 
      You can issue SEND and SEARCH commands for files in your archives, 
      and verify that the server can find them. 

      Terminate this little session with the END command. 

5.6   Adding aliases 

      Create a suitable alias in /usr/lib/aliases (or wherever your 
      aliases are kept): 

            mail-server : "|LIBDIR/listener"

      To get rid of bounced mail, the mail server fakes 'bit-bucket' as 
      its return address. To avoid bounced mails filling your 
      filesystems, add another alias: 

            bit-bucket: /dev/null

5.7   Verification of the installation 

      All the following steps should be executed under the mail server 
      owner! 

      *  Verify your locking strategy. Execute 

            % perl -s testlock.pl -test1 &

         It should say "Got the lock -- waiting ...". Now execute 

            % perl -s testlock.pl -test2 &

         It should say "Good. Could not lock -- waiting ...". Now kill 
         the first process. The second process should print "ret = 1" 
         and exit. 

      *  Verify the working of the 'dorequest' program: 

            % LIBDIR/dorequest <youraddress> <arealfile>

         You should receive mail(s) containing the indicated file. NOTE: 
         If your site is running sendmail, the mail(s) are queued for 
         delivery. They can take some time before they arrive, depending 
         on how often the sendmail queue is run. You may want to inspect 
         the sendmail queue to see if your mail(s) are in it. 

         If no mail(s) arrive, try 

            % LIBDIR/dorequest -debug <youraddress> <arealfile>

         and see what happens. 

      *  Create a text file containing the following lines: 

            From bla
            From: <youraddress>
            
            test
            send HELP
            end

         Now execute the 'process' script by hand: 

            % LIBDIR/process < text-file

         (The program should terminate after processing the 'end' 
         command). Now you should receive a mail (from yourself!) 
         telling you that your request has been processed. 

         NOTE: If your site is running sendmail, and sendmail is 
         configured to use 'queued' delivery, the mail(s) are not 
         delivered immediately. You may want to inspect the sendmail 
         queue to see if your mail is in it, or run the queue by hand. 

         In case of trouble: run LIBDIR/process with "-debug" to find 
         out what happens. 

      *  Execute the 'process' script again with the same input, but 
         remove the line "test" from the text file. 

         Again you should receive a mail (from yourself!) telling that 
         your request has been processed. It should also tell you that 
         your request has been queued. 

         If you configured $auto_runrequest, program 'dorequest' will be 
         run to handle your request. You should receive a mail with the 
         requested file. 

         In case of trouble: run LIBDIR/process with "-debug" to find 
         out what happens. This will prevent 'dorequest' from being 
         invoked, so you can study the contents of LIBDIR/queue. 

      *  Execute the 'listener' program with the same input. 

         Again you should receive a mail message, but this time it 
         should be from the mail server owner. 

         If you configured $auto_runrequest, program 'dorequest' will be 
         run to handle your request. You should receive a mail with the 
         requested file. 

      *  As another user, run the 'listener' program with the same 
         input. 

         Again you should receive a mail, still originating from the 
         mail server owner. 

      *  Still going strong? Now, as another user, send a mail message 
         to your mail server: 

            % mail mail-server
            send foo bar INDEX
            end
            ^D

         (Mail programs usually do not terminate after reading the 'end' 
         line. Issue Control-D or whatever you local EndOfFile setting 
         is). 

         You will receive a return mail, indicating which archive 
         entries were found, and how they would be transmitted. 

      *  If you did not configure $auto_runrequest, change back to the 
         mail server user, and inspect LIBDIR/queue to see that your 
         request is in it. 

         Run LIBDIR/dorequest without arguments; the selected file 
         should be sent to you by email or UUCP. 

      If you receive the requests, your mail server is fully 
      operational! 

      Likewise, you can test the correct functioning of delivery via 
      UUCP. 

5.8   The logfile 

      If you don't want to keep of log of transactions, remove the file 
      LIBDIR/logfile that was created by the Makefile. 

      NOTE: This is strongly discouraged, since it will also disable the 
      delivery of error messages. 

5.9   Helpfiles and other goodies 

      As a service: place the files 'HELP' and 'unpack.pl' in one of 
      your archives. The Makefile prepares PUBDIR (defaults to 
      LIBDIR/pub) for this purpose. 

      Also, supply shar-files with the btoa/atob handling programs, 
      uudecode/uuencode, and compress/uncompress. 

      The contents of file LIBDIR/mserv.notes is prepended to every 
      reply the mail server sends. This file can be used to supply a 
      daily message about the server, new entries, etc. 

      The contents of file LIBDIR/mserv.hints is appended to every reply 
      the mail server sends. 

      Make sure your archives have a decent copy of Dumas uud/uue and 
      xxencode/xxdecode if you intent to support these. Likewise, zoo, 
      zip, etc. 

      Also, provide a file 'INDEX' that more or less describes what is 
      in your archives. People are going to ask for it. 

      Since people usually don't read the documentation, link 'help' to 
      'HELP', 'index' to 'INDEX', 'atob.shar' to 'btoa.shar', etc. 

5.10  Daemon processes 

      Install crontab entries for the mail server processes. A sample 
      crontab is included in the distribution as CRONTAB.sample. 

            30 0,2,4,6,18,20,22 * * * LIBDIR/do_runq
            0 3 * * * LIBDIR/makeindex
            0 7 * * * LIBDIR/do_report -errors -since .errrun
            10 7 * * 7 LIBDIR/do_report -full -collect -ftp -ftpclean

      The above example runs the mail server queue every two hours, 
      except during working hours. Once a day any mail server errors are 
      reported, once a week the logfile data is accumulated and a usage 
      report is generated. The FTP cache is reported and cleaned. 

      CHECK AND MODIFY THESE SCRIPTS SINCE THEY WILL PROBABLY NOT DO 
      WHAT YOU WANT! 

      Every night a three o'clock new index files are generated. You may 
      leave this line out if you do not use index files. 

      If you configured $auto_runrequest in 'ms_common.pl', there is no 
      need to run the queue that often; once or twice a day will be 
      sufficient. 

5.11  Let Me Know 

      If you have implemented the mail server on your system, please let 
      me <jv@mh.nl> know so I can collect information about the 
      implementations. Also let me (and the net!) know what kind of 
      archives you are offering. 

      And protect your and mine freedom to write programs: join the 
      League for Programming Freedom. 

            League for Programming Freedom
            1 Kendall Square #143
            P.O.Box 9171
            Cambridge, Massachusetts 02139
            E-mail: league@prep.ai.mit.edu

6.    ADVANCED TECHNIQUES 

6.1   Extending the Mail Server command set 

      Both the command parsing and execution are implemented using 
      dispatch tables to subroutines. As a result, it is very easy to 
      extend the Mail Server command set. 

      In the configuration file, $cmd_extend can be defined to the name 
      of a (perl) file to load extensions. 

      Mail Server extensions can be implemented as follows: 

      1. Write a subroutine to parse the command. See 'pr_parse.pl' for 
         lots of examples. Any work should be pushed on the @workq. 

      2. Add a command verb to $cmd_tbl, pointing to this routine. The 
         command verb must be in ALL UPPERCASE. 

      3. Write a subroutine to execute the command. See 'pr_dowork.pl' 
         for lots of examples. 

      4. Add a command verb to $exe_tbl, pointing to this routine. Since 
         the Mail Server uses uppercase letters for this purpose, please 
         use a lowercase letter or verb. 

      5. Provide a short help message for the HELP command. 

      As an example, the following code adds the 'REPORT' command to the 
      Mail Server. 

            sub cmd_report {                         # step 1.
                # Check syntax.
                # $cmd is the command verb, upcased.
                # @cmd has the remainder of the command.
                return &errmsg ("Usage: $cmd") unless @cmd == 0;
            
                # Push exe command on work queue.
                push (@workq, &zp ('r'));
            
                # Feedback.
                print STDOUT ("=> Okay\n");
                1;
            }
            
            $cmd_tbl{'REPORT'} = 'cmd_report';       # step 2.
            
            sub exe_report {                         # step 4.
                &do_unix ("$libdir/report -usage");
                1;
            }
            
            $exe_tbl{'r'} = 'exe_report';            # step 4.
            
            &add_help ('REPORT',                     # step 5.
                'Generate a mail server usage report.');

      The source distribution contains a number of extension examples, 
      look for the files ud_sample*.pl. 

      NOTE:  Extending the Mail Server command set is not supported
             and is not garanteed to work in future releases.
             However, when implemented as described above, the impact
             of a new release will probably be small.

6.2   Setting up multiple servers 

      You can set up multiple servers sharing the same software. All 
      mail server programs support the command line option "-config" to 
      specify an alternate configuration file for the program. This file 
      can be used to select different archive directories. Depending on 
      your needs, you can share the logfile, lockfile and queuefile. 
      However, If you wish to share the queuefile, you *NEED* to share 
      the lockfile also. 

      You can have multiple mail aliases, e.g.: 

            mail-server : "|LIBDIR/listener"
            debug-server: "|LIBDIR/listener -trace"
            info-server : "|LIBDIR/listener -config LIBDIR/info_config.pl"

      All I/O to queue and logfile is done while locking the file, so it 
      is not possible to get file corruption. The lockfile is used to 
      prevent multiple invokations of the delivery program, 'dorequest'. 
      But if you configure a common queue, you need to configure a 
      common lockfile too. 

      The delivery program will deliver requests that are entered in the 
      queue. Who is going to receive what is totally determined by the 
      'process' program. So it is perfectly safe to have a single queue 
      for several flavours of the mail server. Once the information is 
      in the queue, it doesn't matter what program is going to deliver 
      it. 

      NOTE: You need to build the listener program version 3.1 or later. 

6.3   Setting up de Mail Server for Interactive Use 

      If the program 'process' (or 'listener') is invoked with its input 
      to a terminal, it enters interactive mode. Commands can be typed 
      in and are executed immediately. Errors are not considered fatal. 
      Alternatively, interactive mode can be forced with the 
      "-interactive" command line option, and prohibited with "-noi". 

      In interactive mode, the program displays a sign-on message, 
      followed by the contents of the $notesfile. However, if a file 
      exists with the name of the $notesfile followed by 'i', this file 
      is used instead. This way you can have different messages for 
      interactive and non-interactive use. 

      In interactive mode, the $hintsfile, normally included at the end 
      of the feedback message, will be appended to the output of the 
      HELP command. If a file exists with the name of the $hintsfile 
      followed by 'i', this file is used instead. 

6.3.1 Via login 

      Add a login name to your system to be used for this purpose. 
      Supply a dummy HOME directory (e.g. /tmp), and specify 
      LIBDIR/listener for login shell. 

6.3.2 Via inetd 

      Choose a port to be used for this service, e.g. 2000. Add this 
      port to /etc/services: 

            mserv   2000/tcp

      Add the following line to the inet configuration file (usually 
      /etc/inetd.conf): 

            mserv    stream tcp nowait LIBDIR/listener listener -i0

      Depending on your system, you may have to insert a user-id after 
      'nowait'. Use whatever is appropriate. The listener program will 
      set UID anyway. 

      The command line option "-i0" forces interactive mode, and 
      attaches standard output and error to standard input (file 
      descriptor 0). 

7.    DESIGN AND MAINTENANCE OF THE ARCHIVES 

      The mail server software can handle multiple archive directories. 
      Every directory specified in @libdirs is treated equal. 

      Please consider the following points: 

      *  Hidden files (filenames starting with '.') and files that are 
         not readable to 'mserv' cannot be retrieved. They might show in 
         the index, however. 

      *  If a file occurs multiple (e.g. 'INDEX') in more than one 
         archive directory, the first occurrence is retrieved. 

      *  If a directory occurs multiple it can not be retrieved. 

      *  For best results: name archives similar to 'foo-XXX.tar.Z', 
         where XXX is a version number, e.g. emacs-18.58.tar.Z or 
         xdvi-12.zoo. This will aid people in finding the right version 
         for a specific archive entry. 

8.    THE MAIL SERVER DISTRIBUTION 

      This section lists the files in the mail server distribution, with 
      a short explanation. Note that lots of functions are contained in 
      separate files. These files are only loaded if needed, cutting 
      down on the system resources. 

      README           A short introduction.
      ChangeLog        A log of changes. Relevant for developers only.
      INSTALL          The ASCII version of this document.
      ms_config.pl     Configuration file.
      ms_common.pl     Common information for the mail server constituents.
      process.pl       Perl script to parse mail messages, and to
                       enqueue the requests.
      pr_*.pl          Demand loadable modules for process.pl
      pr_doindex.pl    Index lookup (for INDEX command).
      pr_dowork.pl     Looking up files.
      pr_dsearch.pl    Directory searching (for SEND commands).
      pr_ftp.pl        FTP operations.
      pr_help.pl       HELP message.
      pr_isearch.pl    Index searching (for SEND commands).
      pr_parse.pl      Command parsing
      rfc822.pl        A package to process rfc822 based messages.
      ftp.pl           Modified version of Alan R. Martello's package
                       to deal with FTP.
      chat2.pl         Modified version of Randal Schwartz' chat2.
      dateconv.pl      Date conversions.
      dorequest.pl     Perl script to encode and split the files, and sent
                       them to the requester.
      dr_*.pl          Demand loadable modules for dorequest.pl
      dr_mail.pl       Sending files via email.
      dr_uucp.pl       Sending files via UUCP.
      dr_pack.pl       Packing directories before sending.
      ms_lock.pl       Portable file locking package
      unpack.pl        Perl script to unpack a concatenated series of parts.
      HELP             The ASCII version of the user guide.
      mlistener.pl     Generator for a simple wrapper around "process.pl"
                       to enable setuid processing.
      makeindex.pl     A simple script to aid in maintaining an index
                       of archive entries.
      chkconfig.pl     A tool to feed back on the configuration file.
      testlock.pl      A tool to test the locking package.
      report.pl        A tool to generate usage and error reports.
      do_report.pl     A script to use report.pl
      do_runq.sh       A shell script to run the mail server queue from cron.
      CRONTAB.sample   A sample cron tab for the mail server.
      mserv.notes      
      mserv.hints      
      mserv.hintsi     Sample files.
      ud_*.pl          Sample Mail Server extensions.
      ud_sample1.pl    'REPORT' command.
      ud_sample2.pl    'SEND CONFIG' command.
      ixlookup.patch   Patch to GNU find 3.5 'locate.c' to create
                       the index lookup program.
      Makefile         To install the package.
      patchlevel.h     Patch version check.
      MANIFEST         List of files.

      Not included but also available are nicely formatted PostScript 
      versions of the User Guide and Installation Guide. To obtain them, 
      send a mail message to <mail-server@nluug.nl> with contents: 

            begin
            send usrguide.ps.Z
            send mservmgr.ps.Z
            end
