


MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



NAME
     mirror - mirror packages on remote sites

SYNOPSIS
     mirror -[dvTn] [-Ufilename [-ppackage] [config-files]
     mirror -[mtfr[G|P]] [-kkey=val] [-cconfig-file]
        [-ssite] [-uuser] {local_dir remote_dir [get_patt]}+

DESCRIPTION
     Mirror is a package written in Perl that uses the ftp proto-
     col  to  copy  directory  hierarchies  between machines.  It
     tries to avoid copying files unnecessarily.

     It is commonly used in one of two ways.  In the first  usage
     given  in the synopsis above, mirror is used to make a local
     ftp archive mirror the contents of a remote ftp archive.   A
     minimal  number of arguments are required and mirror is con-
     trolled by the settings read from  the  configuration  files
     (or standard input).

     In its second use mirror is controlled by command line argu-
     ments.   It provides an easy way to get, or put, a directory
     heirarchy.

     It was written to mirror remote Un*x archives but has  grown
     (like topsy).

OPTIONS
     -d   Enable debugging.  If this argument is given more  than
          once  the debugging level will increase.  Currently the
          maximum useful level is three.

     -n   Do nothing except compare local and remote directories.
          Sets  debug  level  to two, so you are shown a trace of
          what would be done.

     -T   Force the time stamps of any local files to be reset to
          be  the  same  as  the remote file.  Normally only used
          when initialising a mirror area  using  existing  files
          contents.

     -ppackage
          Only mirror the given  package.   This  option  may  be
          given  multiple times in which case all the given pack-
          ages will  be  mirrored.   Without  this  option  being
          specified all packages will be mirrored.

     -v   Print the version details of mirror and exit.

     -U[filename]
          Record all uploads into filename.  Remeber that  mirror
          changes  into  local_dir  to  do  its work, so filename



Sun Release 4.1    Last change: 13 August 1991                  1






MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



          should be a full pathame.  If no filename is  given  it
          defaults to `pwd`/upload_log.day.month.year.

     -G   Get files from  the  remote  machine.   The  local  and
          remote  directories  have  to  be  given on the command
          line.

     -P   Put files onto  the  remote  machine.   The  local  and
          remote  directories  have  to  be  given on the command
          line.

     -Cfile
          Specify config files.  Needed to give config files with
          -P and -G options.

     -kkey=value
          Override any default key/value.

     -m   Equivalent to -kmode_copy=true

     -t   Equivalent to -ktext_mode=true

     -r   Equivalent to -recursive=false

     -sSite
          Equivalent to -ksite=Site

     -uUser
          Equivalent  to  -kremote_user=User.    You   are   then
          prompted  for  a  password,  with echo turned off.  The
          password is used to set remote_password.

     The configuration file is parsed as a series of  statements.
     Blank  lines  and  lines  beginning with a hash are ignored.
     Each statement is of the form:

     keyword =value
     or
     keyword +value

     You  can  add  whitespace  before  the   keyword   and   the
     equals/plus.     Everything    immediately   following   the
     equals/plus is the value, including any leading or  trailing
     whitespace.   The  equals  version  sets the keyword to this
     value. The plus version concatenates the value onto the  end
     of the default.

     A statement can be continued over multiple lines  by  ending
     all lines except the last, with the character ampersand (&).
     The line following the ampersand it appended to the  current
     line with all leading whitespace is removed.




Sun Release 4.1    Last change: 13 August 1991                  2






MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



     Here is a  list  of  the  keywords  and  their  values,  any
     defaults  are  given  inside square brackets.  Those options
     flagged with a star are not yet implemented.

     package        should be a unique name for the package to be
                    mirrored

     comment        used in reports

     skip           Setting this entry causes this package to  be
                    skipped.   (Its  easier  than  commenting the
                    entry out.)

     site           sitename or ip address of the remote site

     remote_dir     remote directory to mirror (or copy into, for

     local_dir      local directory to copy into (or mirror, for

     remote_user    username to use at remote site [anonymous]

     remote_password
                    password    to    use    at    remote    site
                    [user@localhostname]

     get_patt       regexp of remote pathnames to retrieve [.]

     exclude_patt   regexp of remote pathnames to ignore

     update_local   Set get_patt to be local_dir/*.  This is use-
                    ful  if you only want to mirror selected sub-
                    directories of a remote archive.  [false]

     local_ignore   regexp of local pathnames to totally  ignore.
                    Useful to skip restricted local directories.

     do_deletes     delete local files if not  in  remote.   Does
                    not  delete the file yet.  Just prints a note
                    about what to delete.  [false]

     really_do_deletes
                    delete local files if not  in  remote.   Does
                    it, really!  [false]

     delete_excl    regexp of local pathnames to never delete

     max_days       if >0 ignore files older than this many  days
                    [0]

     split_max      if >0 and size of file greater than this  the
                    file   is  split  up  to  be  stored  locally
                    (filename must also match split_patt) [0]



Sun Release 4.1    Last change: 13 August 1991                  3






MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



     split_chunk    Size  of  chunks  to  split  up  files   into
                    [102400]

     split_patt     regexp of remote pathnames to split up before
                    storing locally

     local_ls_lR_file
                    local file containing ls-lR - else use remote
                    ls_lR_file.   This  is useful when first mir-
                    roring a large package.

     ls_lR_file     remote  file  containing  ls-lR  -  else  run
                    remote ls

     name_mappings  remote to local pathname mappings (a  perl  s
                    command,  eg  s:old:new:)  currently only one
                    allowed

     get_newer      get the remote file if its date is newer than
                    local [true]

     get_size_change
                    get the file if size if different from local.
                    If a file is compressed when fetched then the
                    size is automatically ignored.  [true]

     compress_patt  regexp of files to  compress  before  storing
                    locally.  (See also get_size_change.)

     compress_excl  regexp of files not to compress [\.[zZ]]

     force_times    Force local times to match remote times [yes]

     retry_call     If initial connect fails retry ONCE after ONE
                    minute [yes]

     update_log     Filename, relative  to  local_dir,  where  an
                    update report is to be kept

     mail_to        Mail a report to this comma separated list of
                    people

     user           User name or uid to give to local pathnames

     group          Group name or gid to give to local pathnames

     file_mode      Mode to give files created locally [0444]

     dir_mode       mode  to  give  directories  created  locally
                    [0755]

     timeout        timeout ftp requests after this many  seconds



Sun Release 4.1    Last change: 13 August 1991                  4






MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



                    [20]

     ftp_port       port number of remote ftp daemon [21]

     proxy          set to 1 to use proxy ftp service [0]

     proxy_ftp_port port  number  of  proxy-service  ftp   daemon
                    [4514]

     proxy_gateway  name of proxy-service, may also  be  supplied
                    by   environmental   variable   INTERNET_HOST
                    [internet-gateway]

     recursive      do sub directories as well [true]

     flags_recursive
                    flags to send to ls to do a recursive listing

     flags_nonrecursive
                    flags to send to ls  to  do  a  non-recursive
                    listing

     mode_copy      flag indicating if we need to copy  the  mode
                    bits [false]

     interactive    noninteractive copy default [false]

     text_mode      transfer in binary mode by default [false]

     force          transfer selectively by default [false]

     getfile        perform get, not put by default [true]

     verbose        Verbose messages [false]

     disconnect     disconnect from remote site at end of package
                    [false]

     *remote_fs     Remote file store  type.   (Only  copes  with
                    unix at the moment) [unix]

     mail_prog      Program called to send to the  mail_to  list.
                    [mail]

     delete_source  Delete  the  source  files  and   dirs   once
                    transfered.  [false]

     Each group of keywords defines how to  mirror  a  particular
     package  and  should  begin with a unique package line.  The
     package name is used in report  generation  and  by  the  -p
     argument,  so  pick  something mnemonic.  The minimum needed
     for each package is the site, remote_dir and local_dir .  On



Sun Release 4.1    Last change: 13 August 1991                  5






MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



     finding a package line all the default values are reset.

     If the package name is defaults then no  site  is  contacted
     but  the  default values given for any keywords are changed.
     Personally I begin my config files with:

          package=defaults
               remote_password=ukuug-soft@doc.ic.ac.uk
               get_newer=yes
               get_size_change=yes

     If the package is not defaults then mirror will perform  the
     following steps.  Unless an internal failure is detected any
     error will cause the current package to be skipped  and  the
     next one tried.

     If Mirror is not already  connected  to  the  site  it  will
     disconnect  from  any  site  it is already connected to then
     attempt to connect to the remote site's ftp daemon.  It will
     then  login  using  the  given remote username and password.
     Once connected mirror turns on binary mode transfers.   Next
     it  changes to the given local directory and scans it to get
     the details of the local files that already exist, if neces-
     sary the local directory will be created.  Once this is com-
     pleted the remote directory is  similarly  scanned.   Mirror
     does this by changing to the remote directory and runing the
     ftp LIST command, passing the -lRt options.  (I am not  very
     happy  about  this  bit  and hope to allow it to pull back a
     file containing the remote directory listing instead.)  Each
     remote  pathname  will have any specified mappings performed
     on it to create a local pathname.  Then any checks specified
     by the exclude_patt, max_days, get_newer and get_size_change
     keywords are applied on names of files  or  symlinks.   Only
     exclude_patt checking is applied to directories.

     The above creates a list of all required  remote  files  and
     the local path names to store them in.

     Once the directory listing is completed all  required  files
     are  fetched  from  the  remote  site  into their local path
     names.  This is done by pulling the file  into  a  temporary
     file  in  the  target  directory.  If required the temporary
     file is compressed.  The temporary file is renamed when  the
     transfer is successful.

EXAMPLES
     Here  is  the  mirror.defaults  file  from  the  archive  on
     src.doc.ic.ac.uk.

          # This is the defaul mirror settings used by my site:
          # src.doc.ic.ac.uk (146.169.3.7)
          # This is home of the UKUUG Software Distribution Service



Sun Release 4.1    Last change: 13 August 1991                  6






MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



          #
          # Lee McLoughlin <lmjm@doc.ic.ac.uk>

          # Set my defaults
          package=defaults
               # Keep all local_dirs relative to here
               local_dir=/vol/public/
               remote_password=ukuug-soft@doc.ic.ac.uk
               mail_to=lmjm
               dir_mode=0755
               file_mode=0444
               user=0
               group=0
               get_newer=yes
               get_size_change=yes
               # Don't overwrite my mirror log with the remote one.
               # Don't pull back any of their mirror temporary files.
               exclude_patt=^.mirror$|^MIRROR.LOG$|^.in..*.$|^#.*#|^lost+found/
               # Don't compress arc, zip, boo, readme files and index.txt files
               compress_excl+|.arc$|.zip$|.boo$|[Rr][Ee][Aa][Dd][Mm][Ee]|index.txt
               # Keep a log file in each updated directory
               update_log=.mirror
     And here is part of the mirror.config:

          package=gnu
               comment=Powerful and free Un*x utilities
               site=prep.ai.mit.edu
               remote_dir=/pub/gnu
               local_dir+gnu
               exclude_patt+|^ListArchives/|^lost+found/|^scheme-7.0/|^.history
               # I tend to only keep the lastest couple of versions of things
               # this stops mirror from re-pulling the older versions I've removed
               max_days=30

          package=elisp-archive
               site=tut.cis.ohio-state.edu
               remote_dir=/pub/gnu/emacs/elisp-archive
               local_dir+gnu/EmacsBits/elisp-archive

          package=X
               comment=The X Area at export
               site=export.lcs.mit.edu
               remote_dir=/contrib
               local_dir+X/contrib
               # go-1.0.b.tar.Z is immense so I store it split locally.
               exclude_patt+|^unicom|^go-1.0.b.tar.Z
               # I tend to only keep the lastest couple of versions of things
               # this stops mirror from re-pulling the older versions I've removed
               max_days=30

          package=cnews
               comment=The C News system



Sun Release 4.1    Last change: 13 August 1991                  7






MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



               site=ftp.cs.toronto.edu
               remote_dir=/pub/c-news
               local_dir+news/c
               compress_excl+|patches/PATCHDATES
               compress_patt=patches/
               exclude_patt+|^c-news.Z

          # and on, and on ...

HINTS
     Always on adding in a new package check it out first turning
     on the -n option.

     If you are adding to an existing archive then it is ususally
     best to force the timestamps so time comparisions will work.

     Try and have all packages that are being retrieved from  the
     same  site  one  after the other.  That way mirror will only
     have to login once.

     Keep your default settings in a separate file.  That way you
     will,  hopefully,  be able to share mirror details with oth-
     ers.

NETIQUETTE
     If you are going to mirror a remote  site  please  obey  any
     restrictions  that  the site administrators place on access.
     You can generally find the restrictions on  connecting  into
     the  archive  using  the standard ftp command.  Any restric-
     tions are normally given as a login banner or  in  a,  hope-
     fully, obvious file.

     Here are, what I hope are, some good general rules.

     Only mirror a site well outside the working  hours  of  both
     the local and remote sites.

     It is probably unfriendly to try to  mirror  a  remote  site
     more than once a day.

     Before trying to mirror a remote site try and find the pack-
     ages you want from local archives, no one will be pleased if
     you soak up a lot of network bandwidth needlessly.

     If you have a local archive then tell  people  about  it  so
     they  don't  have  to  waste bandwidth and CPU at the remote
     site.

     Do remember to check your config files from time to time  in
     case  the  remote  archive has changed their access restric-
     tions.




Sun Release 4.1    Last change: 13 August 1991                  8






MIRROR(1L)        MISC. REFERENCE MANUAL PAGES         MIRROR(1L)



     Check the remote site regularly for any new restrictions.

SEE ALSO
     perl(l), ftp(1)

BUGS
     The remaining keywords need to be implemented.

     Should be able to mirror non Un*x sites (it may be  able  to
     but I have not tested this - the remote ls is the problem).

     It should restart file transfers where they left off.

     Hanging data transfers should be detected.

     Should allow for multiple packages from the same host, effi-
     ciently.

     Some of the netiquette guidelines should be enforced.

     Should be able to cope with links as well as symlinks.

     Beginning to suffer from creeping featurism.

AUTHOR
     Written by Lee McLoughlin <lmjm@doc.ic.ac.uk>.  It uses  the
     ftp.pl  package  by: Alan R. Martello <al@ee.pitt.edu> which
     uses  the  chat2.pl   package   by:   Randal   L.   Schwartz
     <merlyn@iwarp.intel.com>


























Sun Release 4.1    Last change: 13 August 1991                  9



