# refdbd administration

This chapter will explain the usage of refdba for the administration of refdbd. Please refer to the reference chapter about refdba to learn about command-line switches and interactive commands of this application.

## Create a database

Create new databases with the refdba command `createdb foo`. The new database will be empty initially. If you start adding datasets, the first dataset will have the ID 1.

In order to access the new database, users need to have their access rights set accordingly.

Due to restrictions of the database servers and of the way database names are encoded in refdb SGML/XML/LaTeX documents, the names of the databases should be treated as case-insensitive and should not contain colons (':').

Some database engines support different character encodings. In this case you can use the `-E` option to select an appropriate encoding. See the documentation of your database engine for available encodings.

Your SQL database server may manage more databases than your refdb databases. In this case it may be prudent to use a common prefix for all refdb reference databases that you create. By providing a simple regular expression to the `listdb` command, the execution time may drop considerably.

## Delete a database

Delete existing databases with the refdba command `deletedb foo`.

The contents of the database as well as the database structure will be gone completely. This is different from deleting all datasets. The latter will leave the database structure intact so you can add references again. The former will force you to create a new database before you can add references again.

Usually this command also removes any permissions to the given database that you have granted to users. If you re-create a database with the same name later, you will have to grant permissions again.

## Add or remove a user

### Add users

Adding a user means to allow a user, identified by a database user name and optionally by a password, to read and to modify a particular refdb database.

Both the database user name and the database password may be different from the login name and login password used to get access to the operating system.

Some database engines like SQLite do not support access control. There is no need to add or remove users with these database engines.

refdb uses the access control facilities of the database server. Therefore, the details of adding a user differ slightly between MySQL and PostgreSQL. In any case, refdb is designed to simplify user management as far as possible. This is mainly achieved by combining the information necessary to authenticate a user and the information necessary to grant access to specific database objects into a single command, the `adduser` command of refdba. This command also silently takes care of the fact that users need access to an internal refdb database in addition to the specified reference database.

Database servers use the username, optionally a password, and the host a user tries to connect from to authenticate a user. From the database server's point of view, refdbd is the database client that actually connects to the server, not the refdb clients refdba, refdbc, and refdbib. Database servers usually distinguish between local connections (i.e. by users logged into the same computer) and remote connections (i.e. all connections via TCP/IP). If both refdbd and your database server run on the same computer, all connections will be local and the host-based access control is fairly simple, as both MySQL and PostgreSQL allow local connections by default. All you need to do is:

    refdba: adduser -d foo jack -W newpass

This will allow the new user "jack" to connect to the database server and to access the contents of the database "foo". "jack" has to identify himself by providing the password "newpass" when starting a refdb client.

If a user is already known to the database server, e.g. if he already has access to a different database, you don't have to specify the password again. If you do provide a password, the password of that particular user will be changed to the new one.

Things get a tad more complicated if refdbd and your database server do not run on the same computer. As this is where MySQL and PostgreSQL differ, we'll look at these two cases separately.

#### MySQL

On many operating system distributions, MySQL is configured to accept only local connections. Either the mysqld process is started with the `--skip-networking` option, or the `my.conf` configuration file contains the corresponding option `skip-networking`. In order to allow remote connections, please remove these options from your system.

MySQL allows to alter the host-based component of access control through the SQL interface. The `adduser` command has an additional option `-H` to specify the host or the network where refdbd runs:

    refdba: adduser -d foo -H % jack -W newpass

In this example, refdbd may run on any host. You can as well specify a subnet (-H 192.168.1.%) or one specific host (-H mono.mycomp.com).

#### PostgreSQL

By default, PostgreSQL will accept only local connections. To allow remote connections, the postmaster process must be started with the `-i` option. Change your start scripts accordingly.

The host-based component of access control is not available through the SQL interface in this database server. The `-H` option of the `adduser` command is therefore ignored. Instead the database administrator has to edit the configuration file `/home/pgsql/data/pg_hba.conf`. The following entries would:

- Allow local access (i.e. refdbd and postmaster run on the same computer) to the databases refs and refdb. The latter is an internal refdb database that users must be able to access. Users must provide a password.

- Allow access to the databases refs and refdb through refdbd instances that run somewhere in the network "192.168.1.0". Users must provide a password.

    #   host  DBNAME  IP_ADDRESS  ADDRESS_MASK  AUTH_TYPE  [AUTH_ARGUMENT]
    local refdb                            crypt
    local refs                              crypt

    host  refdb  192.168.1.0 255.255.255.0 crypt
    host  refs    192.168.1.0 255.255.255.0 crypt

The default `pg_hba.conf` file allows local access to all databases without password protection. This may not be what you want. The file contains a bunch of helpful comments, though. You may also want to peruse the PostgreSQL documentation for more information about host-based access control.

### Remove users

This is done with the deleteuser command. The access rights will be revoked for the specified username and database. No other access rights will be modified. The following command will revoke the access rights of user "jack" on the database "foo".

    refdba: deleteuser -d foo jack

Keep in mind that user "jack" still can access the refdb main database `refdb` and any other databases he was granted access to.

If you use MySQL as your database server and refdbd runs on a different box than mysqld, you also have to specify the host or network with the `-H` option, just like when you added the user in the first place. See the Add users section for the specifics.

## Configure the application server

There are several ways to configure the application server refdbd to your needs. Two of these ways will modify the behaviour at startup, while the remaining two ways will reconfigure the server only for the duration of the current run.

### Configuration at startup

There are two ways to permanently change the refdbd configuration. First, you can edit the configuration file which holds the default values. Second, you can modify the command line parameters in the script that starts refdbd in the first place (e.g. an init.d script).

### Reconfiguring a running refdbd process

Again there are two ways to reconfigure an already running refdbd server. The recommended way is to send a HANGUP signal to the server (`kill -HUP pid`) after editing the configuration file. You can send the signal either manually or use the refdbctl script instead. This of course requires that you have a shell on the machine running refdbd. This should not be a problem if you sit in front of that machine. If you don't, and you can't do a remote login (e.g. because it is not supported, as on Windows), this won't help. In that case you may use the second possibility. refdb supports remote administration via refdba if refdbd is started with the appropriate parameter. This allows you to use the refdba `confserv` command to reconfigure the server.

The current implementation of the access control to use the `confserv` command requires the refdb administrator to be a database server administrator. This is different from being root on the machine.

You can change the settings for the timeout and the settings for the logging facility.

    refdba: confserv timeout 60

This will set the timeout to 60 s, i.e. a client-server connection has to be stalled for 60 seconds before refdbd gives up.

    refdba: confserv loglevel 7

This will set the log level to 7.

    refdba: confserv logdest 1

This will set syslog as the log info destination.

    refdba: confserv logfile /var/log/mylog

This will use `/var/log/mylog` as logfile if logdest is set appropriately.

## Add, edit, or delete bibliography styles

This task is currently restricted to the system administrators in order to prevent users overwriting each others bibliography styles. A more appropriate solution will be implemented later to allow users uploading their own styles.

Use the refdba command `addstyle` to add one or more bibliography style files to the database. Existing styles with the same name will be overwritten without a warning, so you should check with the liststyle command before you go ahead.

The name of a bibliography style is encoded in the XML file. The filename of the bibliography style is irrelevant here. Use `grep STYLENAME` to see which bibliography styles are defined in a file and check with the liststyle command whether they're already defined in the database.

refdb uses a non-validating parser to load bibliography styles into the database. It is your responsibility to validate the files before you add them. Adding invalid styles may have undesired and unpredictable results.

The refdba command `getstyle` retrieves one or more bibliography styles as XML documents. In order to edit an existing bibliography style, simply retrieve the style with the `getstyle` command, edit the XML document as necessary, and add it again with the `addstyle` command. As stated above, this will overwrite the existing style definition.

Use the refdba command `deletestyle` to remove one or more bibliography styles from the database. The argument for this command is a Unix regular expression. All styles whose name match the regular expression will be deleted.

## Logging data

The refdb programs can spill out quite a lot of log messages to keep track of what is happening in your programs. This section explains the basics of setting up your message logging.

### What to log

The application server refdbd as well as all command-line clients can generate log messages. Message logging is most important for two purposes:

- Keep track of non-interactive programs. This includes refdbd as well as all client-side tools if they are run from scripts.

- Track down bugs or user errors.

A useful approach is to log all messages with a log level (explained below) of 6. This would give you a good overview over the usage of these programs but would not clutter the log files with debug information. Switch to log level 7 only if you suspect a bug or some user error and need the full debug information to understand the problem.

For the interactive use of the clients logging is usually not necessary, so you'd use a log level of -1 to prevent logging altogether or a log level of maybe 3 or 4. In the latter case you'd get log messages only if something goes badly wrong. Again, if you encounter bugs or user errors you may switch on debug messages by using a log level of 7.

### Destinations

There are three possible destinations the log messages can be sent to. Select the proper destination with either the `logdest` variable in the configuration file or the `-e` switch on the command line. In both cases the values in the following list are accepted. You may use either the numerical value or the case-insensitive string in brackets, e.g. `-e 1` and `-e syslog` are equivalent.

0 (stderr)

  stderr is mostly useful for debugging purposes when refdbd is run as a standalone process and when clients are run interactively. stderr does not make much sense if you run refdbd as a daemon (daemons detach from the console at startup, so all output to stderr is lost).

1 (syslog)

  Sending the data to syslog integrates the log data with the rest of your system's log output. The log messages will be sent to the user facility, which usually is configured to write to `/var/log/user.log`. See the syslog(8) man page for information how to configure the syslog facility.

  If you run refdbd on Cygwin, the syslog messages are sent to the application message list in the NT message logging system. If you are used to going through your log output with tools like grep and awk, you may find it more useful to write to a custom log file instead.

2 (file)

  You can define the full path of a custom log file with either the `logfile` configuration file parameter or the `-L` command line option. By default, refdb applications write their log output to `/var/log/<appname>.log`. Make sure to set the appropriate access rights for these log files.

Some libraries used by refdb, e.g. the PostgreSQL client library, send log messages to stderr. This is not controlled by the log settings of refdbd. The same PostgreSQL log messages show up in the PostgreSQL log which is usually sent to `/var/log/pgsql`.

### Log levels

You can select how verbose the log output of refdbd will be. You can do this by either setting the `loglevel` configuration file variable or the `-l` command line option to an appropriate value. You may either use the numerical value or the case-insensitive string as explained in the table below. E.g. `-l 5` and `-l notice` are equivalent. Set the log level to -1 to disable logging completely. Use a value from 0 through 7 to generate increasingly verbose log output. The definitions of the log levels are taken from the include file `syslog.h`:

Log level definitions

level | verbose | explanation
--- | --- | ---
0 | emerg | system is unusable
1 | alert | action must be taken immediately
2 | crit | the system is in a critical condition
3 | err | there is an error condition
4 | warning | there is a warning condition
5 | notice | a normal but significant condition
6 | info | a purely informational message
7 | debug | messages generated to debug the application

Setting the log level to a given value means that all messages with a priority level up to the given value will be logged. E.g. if you set the log level to 6 (which is a reasonable default value), all messages with a priority from 0 through 6 will be logged, whereas messages with a priority level of 7 will be ignored.

Use log level 7 with caution. The amount of log messages is considerable and sufficient to slow down the application. You should not use this level in everyday use, only to track down bugs or user errors that you may encounter.

### Interpreting the log information

If the messages are sent to stderr, only the message proper will be printed. If the messages are sent to the syslog facility, the default format including the process name and process ID is used.

A line in the custom log file looks like this:

    6:pid=267:Sun Jul 22 18:39:52 2001:application server started

You will easily recognize the following fields:

    message_priority:process_id:time:message

message priority

  This is the priority assigned to the message by refdbd. The values are explained above.

process ID

  This is the process ID (as seen in the `ps ax` listing) of the refdb process that generated the message. In the case of refdbd this may either be the parent process (the one that generated the above "application server started" message) or one of the children that are forked off to answer client requests.

time

  This is the full time and date information when the message was generated.

message

  This is the message text proper.

The message entries are sufficiently structured to allow easy access to the information with the standard Unix tools like awk and grep.

## Security issues

This section briefly discusses some security-related issues that you might want to think about as an administrator. We'll look at the access control provided by the external database servers MySQL and PostgreSQL.

The embedded database engine SQLite does not provide built-in access control. All you can do is use `chown` and `chmod` to restrict access to the database files. There is no way to restrict access through refdb.

### Passwords

refdb tries to support the security features of the SQL database servers as far as possible. This includes the username/password-based access rights scheme (not much surprise here). Since version 0.6.0 the passwords are no longer transmitted as plain text between the clients and the server (but you can request the old behaviour using the `-x` command line options since version 1.0). This means that it now makes sense to keep the passwords secret. There are several ways to specify the password when starting a refdb client. These ways differ with respect to the security of the passwords and are listed here in the order of increasing security:

Specify the password on the command line

  The password is stored nowhere on the filesystem and thus pretty secure from this point of view. But the full command line can be viewed with the `ps` command by any user on the system, so the unencrypted password is basically world-readable at least for a very brief period until the applications have a chance to hide the string.

Store the password in the personal configuration file

  This way the password is protected from other users who habitually run the `ps` command just for the heck of it. But now it is stored unencrypted on the hard drive, and you must make sure that no one else can read the configuration file (no group or world read access).

Specify the password interactively

  This is the default behaviour if the password is not specified either in the configuration file or on the command line. The refdb client will ask for the password. This is certainly the most secure way to provide a password, but this won't work if you run the clients unattended via scripts.

### Database server access control

It is beyond the scope of this manual to reiterate the security models of the database servers, but you need to keep in mind a few aspects relevant to refdb.

- One component of the database server access control is based on the host from which you connect to the database server. This is partially circumvented by the refdb three-tier design. Keep in mind that only refdbd communicates with the database server. Therefore only the host where refdbd runs is relevant for the access control. There is currently no system in place for checking whether a client is allowed to connect to the refdbd application server from a particular host.

- Both MySQL and PostgreSQL distinguish between local and remote connections. Access needs to be granted separately if you want to use both local and remote connections

  On many operating systems the default installations of MySQL and PostreSQL do not allow remote connections for security reasons. You need to manually allow remote connections as described below.

- The refdba commands to remotely reconfigure a running application server are currently protected by a simple table access test. In any serious database server installation, only the database administators have read access to certain system tables. The current implementation of this check requires that you have access to this table if you want to run the `confserv` commands. You should be aware that if the access rights are set up improperly, you may also allow everyone and their grandma to stop or reconfigure the refdbd application server. If you cannot restrict read access to system tables for whatever reason, you should not enable refdbd remote administration (default is off) and use the `kill` command or the refdbctl script instead.

- Most of the refdba commands require database administrator rights.

#### MySQL

If you run MySQL as your database server, these things apply as well:

- When adding users with the `adduser` command the `-H` option has to specify the box where refdbd runs as the host, not the box from where the user will run the clients. If you do not specify a host, "localhost" is assumed. This works on some platforms (e.g. FreeBSD 5.4), but not on others (e.g. Fedora Core 4). In the latter case you may have to specify the hostname explicitly.

- To allow remote connections to the database server, the MySQL configuration file `my.cnf` must not contain the option "skip-networking", and the start script must not use the command-line option `--skip-networking`. Many operating systems use one of these methods in default installations to restrict access to local users for security reasons.

- If you install and run MySQL yourself, you should not use the default database administrator account. The name (root) is widely known and by default this account is not password-protected. To make your database server secure, please create a new database administrator account with a different name and specify a password.

#### PostgreSQL

If you run PostgreSQL as your database server, these hints are for you:

- The host-based part of the PostgreSQL access control is not accessible through the SQL interface. Therefore, the refdba command `adduser` cannot provide the host information (in other words, the `-H` is ignored).

  Instead, the host-based part of the access control is specified in the PostgreSQL configuration file, usually `/home/pgsql/data/pg_hba.conf`. On most operating systems, the default configuration allows unrestricted access for all users on the local system, but no remote access. Use something like the following entries to enforce usage of passwords for both local access and remote access from the network 192.168.1.0 to refdb and a reference database "refs":

      #   host  DBNAME  IP_ADDRESS  ADDRESS_MASK  AUTH_TYPE  [AUTH_ARGUMENT]
      local refdb                            crypt
      local refs                              crypt

      host  refdb  192.168.1.0 255.255.255.0 crypt
      host  refs    192.168.1.0 255.255.255.0 crypt

  As mentioned previously, only the host where refdbd runs is relevant for the host or network entries in the configuration file.

  Make sure to read the PostgreSQL documentation to get your access control right.

  Please keep in mind that `postmaster` (the PostgreSQL database server parent process) needs to be started with the `-i` option to accept remote connections at all. Most default installations do not use this switch to increase security.

- If you install and run PostgreSQL yourself, you should not use the default database administrator account. The name (pgsql) is widely known and by default this account is not password-protected. To make your database server secure, please create a new database administrator account with a different name and specify a password.

## How to run several refdb instances

In all but a few cases, having one refdbd daemon per network is absolutely sufficient. However, there are a few cases where you may end up running more than one instance of refdbd on the same computer at the same time:

- You want to provide access to more than one database engine. The only good reason to do this may be to test-drive a different database engine.

- You want to test-drive a new refdb release without interrupting the access to your installed version

- You do not want to run refdbd as a daemon serving all users. Instead, each user should be able to run his own copy of refdbd

You'll have to configure each refdbd process individually. If each user starts his own copy of the application server, private `.refdbdrc` configuration files in the users' `$HOME` directory allow a simple setup. If you want to run more than one daemon as non-user-processes, you cannot safely use the `refdbctl` script to control the instances. Instead you should start and stop the daemons manually and use the appropriate command-line options to configure each process individually. The options which need to be set differently whenever more than one refdbd instance runs on the same box are:

- `-D`, `-i`, and`-b` to set the database engine, IP address, and the port it listens on, respectively. Only required if you fiddle with two different database engines.

- `-e` and -L to use separate log files for the two instances. This may be easier to evaluate than using syslog, although the processes can be distinguished by means of their process ID.

- `-P` to use separate PID files

- `-p` to set the port each refdbd instance listens on. The ports must be different, otherwise clients can't select which instance to connect to.

On the client side you'll need only set the `port` configuration variable or use the `-p` command line option to select the application server instance that you want to connect to.

## Backup your data

If your database crashes for whatever reasons, the first question you'll hear is: "You've got a backup, don't you?". At this point in your life you shouldn't even think about not having a backup.

There's basically two ways to create a backup of your reference data: Use a risx dump or use a SQL dump. The risx dump is slower but it would allow you to transfer the data to a refdb installation using a different database server. The SQL dump is faster but you're tied to the database server you're using. If you use SQLite as your database backend, you may simply grab copies of the database files. Each database is stored in one file. This file holds all required information and is platform-independent, so it is suitable for backup purposes.

Backups are useful only if you create them regularly. You should work out a schedule based on how often the contents of the database change. Think about setting up a `cron` job for nightly or weekly snapshots.

### Creating a risx dump

All you need to do is to retrieve all references and write them to a file in risx format. Use the `getref` command in refdbc for this purpose:

    	refdbc:
    	getref -t risx -s NOHOLES -o all.ris :ID:>0

The `-s NOHOLES` option will dump skeleton references for references that you deleted from the database. This simplifies re-creating the same ID values in a new database.

### Creating a SQL dump

This is best done with the command line tools shipped with you database server. You should consult the manual of your database server, but the following commands should be all it takes:

If you run MySQL as your database server, run:

    	$
    	mysqldump -u root -p --opt dbname > dbname.dump

This assumes that "root" is the name of your database administrator account and "dbname" is the name of your database.

If you run PostgreSQL instead, use this command:

    	$
    	pg_dump -u pgsql -C dbname > dbname.dump

This assumes that "pgsql" is the name of your database administrator account and "dbname" is the name of your database.

To restore a database from a dump, use the command line clients shipped with your database server. For MySQL, the required sequence is:

    	$
    	mysql -u root -p -e "CREATE DATABASE dbname"

    	$
    	mysql -u root -p dbname < dbname.dump

whereas the following would do the trick with PostgreSQL:

    	$
    	psql -u pgsql template1 < dbname.dump

SQL dumps are also well suited to create backups of refdb, the common refdb database.