# Using RefDB in your programs

## General thoughts

RefDB provides command-line clients to access the application server refdbd. This makes it easy to automate tasks through shell scripts or Makefiles. RefDB uses both possibilities for its own purposes. The prime example for a shell script is runbib, which transform your XML documents by invoking, among others, the RefDB bibliography client. Makefiles are more appropriate in other contexts. The Makefile created by refdbnd is just one example.

However, if you want more than that, you'll want to access refdbd from your own programs. Need RefDB support for your favourite editor? It is doable, as you can see from the RefDB integration in Emacs and in Vim. Want to access RefDB through the web? There is a PHP implementation of a web front-end in the RefDB sources with plenty of room for improvements, and there is also a SRU interface. Want to have a pointy-clicky frontend so you'll no longer have to envy the Mac and Windoze users of EndNote? Well, this is a challenge that no one has tried yet. Before you fire up your favourite IDE and start coding, please consider the following:

Do not even think about fiddling with the SQL database!

Most languages offer a simple way to access SQL databases, and the database engines often provide client libraries for a variety of languages. However, RefDB is more than just a thin layer on top of a SQL database. RefDB normalizes data. RefDB takes data apart and reassembles them when needed. RefDB takes care of character encoding conversions. Even worse, the SQL table schemas may change from one RefDB version to the next. Therefore you should never attempt to access RefDB data from your own programs on the SQL level. Instead, you should rely on the interfaces that RefDB offers.

## Interfaces

There are two interfaces available which will be explained below.

### Call the C clients from your program

Using system calls is fairly straightforward. Most programming languages provide constructs to run external programs and to send data to their standard input or to read data from their standard output (rarely, however, both at a time). The following commands give an example how to accomplish this in Perl:

    $output = `refdbc -d $dbname -C whichdb -u $user -w $pass 2>/dev/null`;
    $result = `refdbc -d $dbname -C addref -t $type -u $user -w $pass $infile 2>&1`;

The first call discards the standard error of the command and writes only the standard output to the variable `$output`. The second call collects both standard error and standard output in a single variable. Your program would have to parse `$result` in order to separate both. In both cases Perl variables are used to supply database names, file names, or the reference type.

This kind of access works from most programming languages. For an example written in Lisp, see the Emacs frontend for RefDB. Easy as it may be, running the C clients from your program does have some drawbacks:

- As shown in the examples above, the handling of standard output and standard error is less than perfect. You can retrieve either, or both in a single variable. Your program therefore needs the logic to separate output from error messages. If you need both separately, you can redirect both to separate files and then read in these files, but this is as kludgy as it sounds.

- On most systems you can't send to standard input and read from standard output at the same time (see this Perl script how you can still achieve this). This can be circumvented by either writing the input data to a file (most RefDB commands can read data from files) or by capturing the input data in a file via redirection.

- Last but not least, you need to have the C clients installed along with your own program.

### Directly talk to refdbd

This is a somewhat more ambitious approach, as you have to implement the client part of the client/server dialog described here. The client/server dialog is a plain-text protocol which is fairly easy to implement in just about any language. The only tricky part is the password encryption, but that can be handed over to a little C program called eenc that is part of the RefDB sources. The RefDB project currently has client libraries for two languages which provide an implementation of the client part of the client/server protocol. The Perl client module provides an object-oriented access to refdbd which you can use in your programs without having to know anything about the client/server protocol. There is also a (rudimentary and currently unmaintained) implementation in Ruby.

To give you an impression of how you can use a client library, look at the following Perl example which does approximately the same as the examples above:

    use RefDBClient::Client;

    # create a Client object. You can create as many as you need
    my $client = new RefDBClient::Client;

    # set the initial connection parameters
    $client->set_conninfo("127.0.0.1", "9734", "markus", "pass", "refdbtest",
               "/home/markus/literature", "/usr/local/share/refdb/css/refdb.css");

    # run the whichdb command. We're not interested in the command summary,
    # but read only the data proper into $data
    $client->refdb_whichdb();
    $data = $client->get_data();

    # in order to add references from a RIS file, we first read the file contents
    # into a Risdata object
    my $risdata = new RefDBClient::Risdata;

    $risdata->read_ris("testdata/pubmed.ris");

    # now we send the data to the server. This time we read both the command
    # summary (which the C clients send to stderr) into $summary and the
    # data proper into $data
    $summary = $clientc->refdb_addref(undef, $risdata, "ris", "ISO-8859-1");
    $data = $clientc->get_data();

The advantages and disadvantages of using a client library are as follows:

- You get a clean, object-oriented interface

- No fiddling with standard error or standard output

- However, there may be no client library for your favourite language

## Tips and tricks

This is currently a ragbag of things to consider. It will hopefully be more structured at a later time.

### Paging output

If you run commands like getref that are likely to return more information than fits on a screen, you can either try to parse the result in your program and then display it chunk-wise. You may have to parse the data anyway for a different reason, so this option may come at no extra cost. However, if you just want to display the data, you can also use the server-side paging which is implemented in all "get*" commands. These commands take an optional limit argument. The general format is limit[:offset]. limit is the number of datasets to display, and the optional offset is the number of references to skip at the beginning. That is, asking refdbd to limit the result set to 5:10 will cause it to ignore the first 10 datasets and send datasets 11 through 15. You can use this in a loop to retrieve the datasets in groups of 5 until no more datasets are available. Server-side paging is preferable in all cases where a query may return lots of data, as you don't have to buffer them locally.