
SECONDARIES

This file describes what you have to do to run a secondary copy of the
RIPE database.

First of all you have to make a choice: do you want to run a copy of the
RIPE database where the database is one big file, or do you want to run
this in so-called "SPLIT" mode (recommended but a bit more complicated).
If you run in split mode, every object type has its own file (available
from ftp://ftp.ripe.net/ripe/dbase/split/) and in normal mode, all
objects are in one big file (ftp://ftp.ripe.net/ripe/dbase/ripe.db)

If you decide to run this in "split" mode, see below what you have to
do to get a secondary copy running. If you are NOT running in split
mode, see at B what to do.

A) For a SPLIT RIPE database:

   Make sure you have something like the following in your config
   for this secondary copy:

   DBFILE  RIPE    /wherever/it/is/ripe.db         SPLIT

   As you notice the config file only knows about the basename of the
   database file, it does not know about all the split files. The fact
   that the third argument SPLIT is there will take care of that.

   The filename mentioned here should be the path name where the real
   database will be. During indexing of a new version you will need a
   temporary directory.

   Transfer all gzipped split files with the 2 letter short format
   extension from ftp.ripe.net and put them in a temporary directory
   (make sure you have enough space for database files and indexes). The
   files must be the files that end in the two letter short format for
   the objects that are in the file, ie use ripe.db.in and NOT
   ripe.db.inetnum.

   Ungzip the files and for each of the split database files in the
   temporary directory say:

   indexdb -t targetdirectory filename

   Targetdirectory is the directory where the database files ought to be
   stored according to your particular config file.

   Now you can start a whoisd and your secondary database is running. If
   you already have a whoisd running, it will automatically see the new
   database file and indexes.


B) For NON SPLIT RIPE database

   Make sure you have something like the following in your config
   for this secondary copy:

   DBFILE  RIPE    /wherever/it/is/ripe.db

   The filename mentioned here should be the path name where the real
   database will be. During indexing of a new version you will need a
   temporary directory.

   Transfer the complete RIPE database from ftp.ripe.net and put it in a
   temporary directory. Go to that directory and say:

   indexdb -t targetdirectory ripe.db

   Targetdirectory is the directory where the database files ought to be
   stored according to your particular config file.



REAL TIME MIRRORING


The above mentioned methods describe how to run a secondary that is
always a bit outdated, usuallly a day. The 'syncdb' program provides a
means to keep your mirror site nearly up to date in real time. Note that
the software for this not perfect yet and needs a bit more work (and
time) to maintain.

The program does the following:

The program can be called every morning (from a cronjob) and then run the
whole day (68400). The program will then check the server every 600
seconds if new new updates came in and fetch the data and update the
local database.

Example usage:

syncdb -h whois.ripe.net -s RIPE -u 68400:600

We currently don't accept any smaller interval then 600 seconds for the
mirror sites to avoid overload situations for the whois server.

- General considerations:
  
  - the server is protected by means of an access list. Please provide
    the RIPE NCC <ripe-dbm@ripe.net> with the IP number of your database
    mirror server.

  - This piece of the software is supported but has some loose ends due
    to some not implemented features as the stored/processed attribute.
    It is only recommended for people that are willing to invest some
    time in getting it running and keeping it running
    
  - every update will make a new small file with the update contents.
    Your system might slow down when you have large numbers of these
    files. Therefore you can best run a cronscript every night to ensure
    that older serial update files are (re)moved. A sample script is
    provided in this file.
  
  - Make sure that all the directories mentioned in the config file
    'etc/ripedb.config' and the resyncdb script exist.
  
  - You can find sample scripts at the end of this document for all tasks
    that are described in this document.
   
     
- First of all, you can probably best resync the database by getting an
  ftp copy every weekend/month. The new database files are always available
  from 3.00am MET. You can use the included script for doing the resync.
  Edit the script according to your local environment and mirroring
  software.

- The mirroring process should be started only after you are sure that
  there are no index jobs running anymore. There is no risk in having
  more copies of this program running at the same time because it does
  proper locking, however, you probably get some unwanted mail with
  messages that warn you about this.
  
  You can start the mirror sync procedure every ten minutes after the
  first mirror resyncdb has been done (20 minutes is very safe). Please
  don't make the interval smaller then 10 minutes for now since I really
  have no experience yet on what our whois server can handle right now.

- Stop updating after 21:00 MET in the evening to allow for some
  maintenance work, automatic consistency checks and indexing jobs that
  needs to be done during the night that might give unwanted effects on
  the mirror copies (this is not really needed, but it gives me the
  chance to do some work if needed with affecting you)


Possible problems:

So what happens when something goes wrong:

- you will probably get loads of mails from the cronjob program
  so you might want to use another userid for this then your personal one
  
- make sure the directories with old serials & incoming serials are
  cleaned up once an a while. If you don't do this the performance of you
  server will become considerably worse in time. I will change this
  by making big files with more serial files but that is not ready yet.
  I am sorry for this.
  
  A 'find' that moves/deletes more then one day old files is probably
  what you want. I have included these lines also in my ftp resyncdb script
  but you might want to use them in a separate script that is executed
  every day.

- If your server is rebooted/crashes you might get problems with the
  serial files. No solution yet. Work is progressing. Advise: Before
  rebooting the machine you can best disable the running cronjobs and
  give some time to let the running jobs die.
  
- If the time interval in your cronjob is too small, you might end up
  having two processes running trying to do the real time mirroring. File
  locking is used to avoid this but you might get a mail from cron that
  the update program found another process running.
  
Below are the sample scripts:


#
# crontab
#
# sunday is day 0, last day is day 6
#
# minutes   hours   day-ot-month   month   day-ot-week   command
#---------------------------------------------------------------------------
#
# clean up the serial directories every day
#
0 2 * * * time /home/david/db.backup/bin/cleanserialdirs
#
# do a full resync once a week
#
0 3 * * 6 time /home/david/db.backup/bin/resyncdb
#
#
# keep the backup up to date
#
# start up... be sure that the indexing is not still running
# 
# syncdb will be changed to make sure that this will not happen
# but doesn't support it right now...
#  
0 9 * * * /home/david/db.backup/bin/syncdb -h whois.ripe.net -s RIPE -u 68400:600
#
# end of cron script


=======================

#!/bin/sh

#
# rsyncdb
#
# (sample) script that mirrors the RIPE database,
# indexes the database and puts it in place
#
# make sure that you change the actual mirroring lines according to 
# your local setup. See comments further on in the script
# 

TOPDIR=/home/david/db.backup; export TOPDIR
BINDIR=$TOPDIR/bin; export BINDIR
FTPDIR=/ncc/ftp/ripe/dbase; export FTPDIR
FROMDIR=$FTPDIR/split; export FROMDIR
DATADIR=$TOPDIR/data; export DATADIR
LOCKFILE=$TOPDIR/locks/CLEANDB.LOCK

#
# Make sure that the following directory exists !!!
#
TMPDIR=$DATADIR/new; export TMPDIR
LOG=$TOPDIR/log/index; export LOG

BASESERIALDIR=$TOPDIR/serials; export BASESERIALDIR
SERIALDIR=$BASESERIALDIR/current; export SERIALDIR
OLDSERIALDIR=$BASESERIALDIR/old; export OLDSERIALDIR
INCOMINGSERIALDIR=$BASESERIALDIR/incoming; export INCOMINGSERIALDIR

DATE="/bin/date"; export DATE
TIME="/usr/bin/time"; export TIME

echo Get database from the ftp site \(`$DATE +%y%m%d%t%H:%M:%S`\)

# change the next *two* cp's and call your mirror script instead, 
# please mirror the /usr/local/bin/gzip'ed versions of the database
#
# you should mirror:
# 
# ftp://ftp.ripe.net/ripe/dbase/split/ripe.db.??.gz
#
# and
#
# ftp://ftp.ripe.net/ripe/dbase/RIPE.CURRENTSERIAL
#
# updated databases are every day available from 3.30 AM MET 
#

cp $FROMDIR/ripe.db.??.gz $TMPDIR

# $BINDIR/mirror -p ripe $TOPDIR/etc/mirror.conf

echo Get serial number file from the ftp site \(`$DATE +%y%m%d%t%H:%M:%S`\)

echo -n "Old serial number: "
cat $SERIALDIR/RIPE.CURRENTSERIAL
echo ""

cp $FTPDIR/RIPE.CURRENTSERIAL $TMPDIR

# $BINDIR/mirror -p serial $TOPDIR/etc/mirror.conf

echo -n "New serial number: "
cat $TMPDIR/RIPE.CURRENTSERIAL
echo ""
echo ""

/usr/local/bin/gzip -d $TMPDIR/ripe.db.??.gz

echo Start normal indexing 

for i in $TMPDIR/ripe.db.?? ; do
    
       echo Indexing $i \(`$DATE +%y%m%d%t%H:%M:%S`\) ...
       $TIME $BINDIR/indexdb $i
    
done
echo ""
echo End of indexing 


echo Move databases and indexes to destination directory \(`$DATE +%y%m%d%t%H:%M:%S`\)...

# lock the databases

cd $TMPDIR

for i in ripe.db.?? ; do
    
    if [ -f $LOCKFILE.$i ] ; then
       echo -n "database $i already locked \($LOCKFILE.$i\) by other process: "
       cat $LOCKFILE.$i
       echo ""
       exit
    else
       echo $$ > $LOCKFILE.$i
    fi
    
done

sleep 20

# calculate the new serial numbers

echo +1 > $TMPDIR/calc.$$
cat $TMPDIR/RIPE.CURRENTSERIAL $TMPDIR/calc.$$ | bc > $SERIALDIR/RIPE.OLDESTSERIAL
rm $TMPDIR/calc.$$
mv  $TMPDIR/RIPE.CURRENTSERIAL $SERIALDIR


# move databases and index files

mv $TMPDIR/* $DATADIR

# unlock the databases

cd $TMPDIR

for i in ripe.db.?? ; do
    
    if [ -f $LOCKFILE.$i ] ; then
       rm $LOCKFILE.$i
    else
       echo database lockfile $LOCKFILE.$i doesn\'t exist anymore
       echo ""
       exit
    fi
    
done

echo Do the first update \(`$DATE +%y%m%d%t%H:%M:%S`\) ...

echo -n "Old serial number: "
cat $SERIALDIR/RIPE.CURRENTSERIAL
echo ""

$TIME $BINDIR/syncdb -h whois.ripe.net -s RIPE

echo -n "New serial number: "
cat $SERIALDIR/RIPE.CURRENTSERIAL
echo ""


echo ""
echo Total time:


# end of script


===============================

#!/bin/sh

#
# cleanserialdirs
#
# script moves older serial update files to the old directory
# and removes them when older then 7 days
#

TOPDIR=/home/david/db.backup; export TOPDIR

BASESERIALDIR=$TOPDIR/serials; export BASESERIALDIR
SERIALDIR=$BASESERIALSDIR/current; export SERIALDIR
OLDSERIALDIR=$BASESERIALDIR/old; export OLDSERIALDIR
INCOMINGSERIALDIR=$BASESERIALDIR/incoming; export INCOMINGSERIALDIR

DATE="/bin/date"; export DATE
TIME="/usr/bin/time"; export TIME

echo move old serials to old serial dir \(`$DATE +%y%m%d%t%H:%M:%S`\)

cd $SERIALDIR && $TIME find $SERIALDIR/RIPE.[0-9]* -type f -mtime +1 -exec mv -f {} $OLDSERIALDIR \;

echo clean up old serial dir \(`$DATE +%y%m%d%t%H:%M:%S`\)

cd $OLDSERIALDIR && $TIME find $OLDSERIALDIR/RIPE.[0-9]* -type f -mtime +7 -exec rm {} \;

echo clean up incoming serial dir \(`$DATE +%y%m%d%t%H:%M:%S`\)

cd $INCOMINGSERIALDIR && $TIME find $INCOMINGSERIALDIR/RIPE.[0-9]* -type f -mtime +7 -exec rm {} \;

# end of script

============================