# Reference management

You can manage your references with the refdbc command line client. We will demonstrate the interactive use of refdbc, but feel free to use the non-interactive batch mode instead if you prefer. We'll also use a conservative, least error-prone approach whenever there are several ways to achieve the same goal. E.g. we'll use intermediate files instead of piping so we can verify what we got at every stage. Once you're sure about what you're doing, you can still speed up things.

## Add references

### RIS datasets

Adding references boils down to running `addref` with proper input files. The input files have to be valid RIS files. They may contain one or more RIS datasets.

You can set some fields to default values with the `-g defaultfile` option of the `addref`. This will add the tags given in the defaultfile to the end of every dataset before they are added to the database. This means that these values override any values given in the datasets. This option may be convenient e.g. to set the reprint status of all datasets that you add in one fell swoop instead of editing them individually.

Please be aware that any numerical values of the ID tag are ignored when you add references with the `addref` command. RefDB assigns an automatically generated ID to every new reference, so you can't enforce arbitrary IDs. You can however use the ID tag to provide an alphanumeric citation key, such as "Miller1999". The citation key should consist only of letters and numbers within the 7-bit ASCII range and has to be unique in the database. Remember that the citation key has to be all uppercase if you want to create bibliographies for SGML documents. If you do not specify a citation key, RefDB will generate a unique citation key based on the last name of the first author and the publication year.

If you want to overwrite an existing reference with a given numerical ID or a given citation key, use the `updateref` command instead.

#### Example 1

Let's assume we just downloaded a bunch of references from PubMed (the largest database for biomedical publications). We saved them in the "Medline" format, ending up with several small files containing one or more references each. How do we get them into the database?

- First we convert all references in the files (we prudently named them `pm001.txt`, `pm002.txt` and so on) to the RIS format and write the result into an intermediate file, using the med2ris.pl input filter:

      	      ~#
      	      med2ris.pl -o foo.ris pm*

  We could now open `foo.ris` in our favourite text editor and make a few changes. We could enter additional keywords, provide personal notes (e.g. why we currently find the article worth the paper it is printed on, something that may not be apparent a few months later), or specify the reprint status and availability.

- Then we switch to (or start) the refdbc command line client and type the following command at the prompt to add the references to the database `bar` (use the full path to `foo.ris` if necessary):

      	      refdbc:
      	      addref -d bar foo.ris

#### Example 2

You feel the urge to import MODS data, an XML data format developed by the Library of Congress. RefDB does not support this format natively or via converters at this time, so you'll have to resort to an external tool. Chris Putnam's bibutils package comes to the rescue, as it provides a set of command-line tools to interconvert a number of bibliography data formats. Both RIS and MODS are supported, so we have all it takes. The following command imports the MODS data into your default database in a single step, although you may first want to check the output of the converter before going ahead:

    	      # xml2ris modsdata.xml|refdbc -C addref

### risx datasets

If you prefer the XML data format instead, use the `addref` with the `-A risx` switch. As the data are imported using a non-validating XML parser you should make sure the documents are valid, either by using some nifty feature of your XML editor or by running the document through a suitable tool like `xmllint`.

Each entry in a risx document can specify a numerical ID value and an alphanumeric citation key. The ID is ignored by the import routines. If you want to update or replace an entry by ID, you have to use the `updateref` command instead. The citation key should only use letters and numbers within the 7-bit ASCII range. Remember that the citation key has to be all uppercase if you want to create bibliographies for SGML documents. If you do not specify a citation key, RefDB will automatically assign a citation key based on the first author and the publication year.

#### Example

You've written a risx dataset from scratch and want to import it into your RefDB database. This is what you need to do:

- First you need to make sure that your document is valid against the risx DTD. Unless your XML editor validates your document anyway, you'll have to use an external validator like `onsgmls`, which is a part of the OpenSP suite of SGML tools, or `xmllint`, which is shipped with libxml2.

      	      ~#
      	      onsgmls -s /usr/local/share/refdb/declarations/xml.dcl foo.xml
      	      ~#
      	      xmllint --noout --dtdvalid "http://refdb.sourceforge.net/risx/index.html" foo.xml

  No news is good news in both cases: If your document is valid, the validator will not output anything. If you do get some error messages, go back to your editor and fix the problems.

- Now switch to refdbc and run the following command to add the references in `foo.xml` to the reference database `bar`:

      	      refdbc:
      	      addref -A risx -d bar foo.xml

## Find and view references

The refdbc command `getref` is your friend to find and display references. The query language that is used to retrieve specific references from the database is explained in detail elsewhere. This section offers a few additional hints.

The general idea of the `getref` command is that you specify what you want and you get back a chunk of text. You can select the format of this chunk (e.g. screen, HTML, or DocBook output) and you can specify what you want to do with this chunk. A suitable strategy for most searches works as follows:

- Run your initial query, send the screen output to a pager, and see whether the result is what you need. We use the screen output because it generates the least verbose output and thus is the fastest.

- If necessary, refine your search as discussed shortly until you arrive at the desired list of publications.

- Re-run the previous query, but use the output format and output destination that you need to work with the data. For example, if you want to edit the selected references, you would use the RIS format and write the result to a file with the `-o` option.

refdbc has a uniquely simple way to rerun and modify queries. Many reference database allow you to modify previous queries only by adding new restrictions to the end or removing restrictions from the end of a previous query. refdbc is more flexible. In the interactive mode, refdbc keeps its own history of commands. You can use the up key or the Ctrl p combo to recall the previous query. You can press these keys repeatedly to go further back in the history. Use Ctrl r to incrementally search for a previous query by entering a short string. Adding more characters will make the search more specific. Pressing Ctrl r repeatedly after entering some search string will display other matches further back in the history.

In the batch-mode, you can use the history feature of your shell. In either case you can modify, extend, or restrict any previous query as you see fit.

## Delete references

The deleteref command accepts one or more reference IDs as its argument to specify the references you want to get rid of. This is quite straightforward if you want to delete one or two references. But what if you have a disjoint list of 17 references, e.g. all publications of an author who was convicted to be fraudulent? A useful strategy to delete such references is to save a query with the `getref` command to a file and use it as a list of ID values for a subsequent run of deleteref. This way you can verify by inspecting the file that you will delete the correct references. Use the `-s ID` option to request minimal datasets containing the ID value instead of the citation key in the ID field.

### Example

To stick with the example of the fraudulent author, we assume his name is, as in so many examples, John Doe (apologies to all RefDB users with that name).

- First we select all his publications from the database `bar`, request only the ID, and save the result in an intermediate file:

      	    refdbc:
      	    getref -t ris -s ID -o tobekilled.ris -d bar ":AU:=Doe,J."

  The result is, once again, a file with plain text that we can check and edit to our needs. E.g. if you want to keep one or two publications in the list, just delete their entries in the intermediate file.

- Then we use the deleteref command to actually delete the references associated with our list of IDs:

      	    refdbc:
      	    deleteref -f tobekilled.ris -d bar

## Edit references

RefDB has no built-in reference editor (but see the Emacs and vim support files which allow you to add and update references from within your editor). The general strategy to edit an existing reference is as follows:

- Save the reference(s) that you wish to edit to a file using a command like `getref -o foo.ris -t ris -d bar ":ID:=1717 OR :AU:=^Miller"`. This would retrieve the reference with the ID 1717 as well as all references with a Mr. or Mrs. Miller on the author list. If you prefer, use the risx output format instead.

- Use your favourite text editor to edit the references in the file `foo.ris`. Make sure you don't touch the ID field as this is crucial to match this reference in the next step with the existing copy in the database.

- Use the command `updateref -d bar foo.ris` to update the reference(s) in the database.

## Print references

There are two ways to print references:

- Send the output of the `getref` command to a printer:

      	  refdbc:
      	  getref -c lpr :ID:>0

  You should probably be a bit more selective about the references than shown above unless you want to empty the paper tray of your printer. In any case, this command will print a list of your references in the screen output format. You can use some additional plumbing along these lines to obtain a more beautiful printer output:

      	  refdbc:
      	  getref :ID:>0 -c 'pr -f | fmt -w 70 | lpr'

- The second option is to write the HTML output to a file and use your browser's print capabilities.

## Managing personal reference lists

Personal reference lists mainly serve two purposes:

- If you share your reference database with other users, your default personal reference list records which datasets you've added. Coincidentally, the name of this list is identical to your database username.

- In addition to your default list, you can maintain as many additional lists as you see fit. Use these lists to select references for particular purposes, e.g. when preparing a presentation or if you need to keep track of papers that you still have to read.

So personal reference list are essentially a tool to have your own personal information for a dataset and to create personal, selective views of the database.

### Creating and deleting personal reference lists

Actually there are no commands to create or delete personal reference lists. These lists are created automatically as soon as you add references to them. Now the question is, how does a reference get into your personal reference list? There are three ways to accomplish this:

1. If you add a new reference to the database with the `addref` command, the reference will be automatically added to your default personal reference list.

2. Add an existing reference to your personal reference list with the `pickref` command. This command will not add personal data to the reference. To add a reference to a list other than your default list, use the `-b listname` switch.

3. If you use the `updateref` command to modify the information in an existing reference, this reference will also appear in your default personal reference list. If you use the `-P` switch with this command, the data of this reference common to all users will remain untouched, but your personal information (reprint status, availability, notes) will be added.

Remember that one user can act on behalf of a different user with the `-U` switch, that commands like `addref` support.

Of course there are also ways to get rid of the references that you once found useful. If you share the database with other users, the first choice should always be the `dumpref` command. This just removes your personal information and your association with the reference, while it leaves the common information untouched for other users of the database. Only if you know that no one else is interested in a reference, you should think about using the `deleteref` command. In both cases personal reference lists will be automatically removed if they are empty.

### Accessing references in personal reference lists

You can limit your database search with the `-b listname` switch of the getref command to those references that you added to that particular personal reference list. If you want to access references in your default list, you have to pass your username as an argument to the `-b` switch.

### Advanced use of personal reference lists

The pickref and dumpref commands mentioned above are actually only simplified interfaces to the extended notes feature. Each personal reference list corresponds to one extended note with these particular features:

- The note is initially almost empty. Only the title, the username and the key are set.

- The title and the key are initially identical and have the format <username>-<listname>. The username prefix is used to create a separate namespace for each user.

However, you can treat these notes just like any other note. In particular, you can retrieve the note with the getnote command, edit it by entering a description as the contents or a couple of keywords, and finally update it with the updatenote command. Just keep in mind that this information will be lost if you remove all references from this list, as this will cause the note representing the list to be purged automatically.

## Global edit references

RefDB has no special command for a global edit of all references or a group of references. You can get pretty far though with a default file when updating references and with Unix plumbing.

To change existing text:

- Run a search on the references that you wish to change and pipe the output through sed into a file, like `getref -t ris -d bar -c "sed 's/foo/faa/g' > refs.ris" "search-string"`.

- Use the command `updateref refs.ris` to update the reference(s) in the database.

To add or change fields:

- Run a search on the references that you wish to change, like `getref -t ris -o refs.ris "search-string"`

- Create a `defaultfile` containing the field(s) that you want to change or add.

- Use the command `updateref -g defaultfile refs.ris` to update the reference(s) in the database.

## Create periodical synonyms

Scientific journals are very often cited by standardized abbreviations which help to keep the space required for bibliography listings at a minimum. However, at times you need the full name instead. If you did not add both synonyms when adding the references that use a journal, you can use the updatejo command to add or edit these synonyms.

## Character encoding issues

The 7-bit ASCII character set originally employed by PCs in the days of yore turned out to be insufficient for languages other than English. Reference data may require characters not included in the ASCII character set. The string sorting order may also follow different rules. RefDB supports national character sets as well as Unicode, which is sort of a superset of all national character sets. As a RefDB user and administrator you'll have to deal with character encoding issues at different levels.

### Character encodings of databases

While it is possible to convert the data during import and export (see the following sections), it is still worthwile to spend a few thoughts about the character encoding used by your reference databases. If possible, use an encoding that ensures a suitable string sorting order for your data. Choosing a proper encoding also avoids unnecessary character encoding conversions when importing or exporting data.

The available encodings are limited by your database engine:

SQLite

  SQLite currently supports only ISO-8859-1 (the default) and UTF-8 as a compile-time option. If you install a binary package, it most likely uses ISO-8859-1.

SQLite3

  SQLite3 uses UTF-8 by default. UTF-16 is supported by the database engine, but not by the libdbi library which RefDB uses to access the engine.

MySQL

  This database engine supports a fairly large number of encodings, but versions prior to 4.1 allow only one encoding per server instance. That is, all databases have to use the same character encoding. Please see the MySQL documentation for the growing list of supported encodings

PostgreSQL

  This database engine supports a variety of encodings as a per-database option. That is, all reference databases may use different encodings. Please see the PostgreSQL documentation for a current list of supported encodings.

### Character encodings of imported data

We'll have to distinguish two different sorts of data:

RIS

  This plain-text format does not have a built-in way to declare the character encoding of the data. Instead you have to use the `-E` option of the addref and updateref commands to specify the encoding if it is different from the default (UTF-8).

  Please note that the import filters med2ris, en2ris, and to a limited extent also marc2ris support on-the-fly character encoding conversion.

risx and xnote

  These are XML formats that can use the XML way of declaring the encoding. This is done in the processing instructions, which is the first line in a XML file. Due to a limitation of the parser used for importing XML data, only four encodings are accepted by RefDB: UTF-8, UTF-16, ISO-8859-1, US-ASCII. If your data use a different encoding, use the `iconv` command line utility (usually a part of the libiconv package) to convert your data to one of the accepted encodings.

### Character encodings of exported data

By default, data are exported without a character conversion, i.e. the data will use whatever encoding the database uses. If you want the exported data in a different format, request the encoding with the `-E` option. This option is accepted by the getref and getnote commands of refdbc as well as by the refdbib client. You may request any encoding that your local libiconv installation supports. `man 3 iconv` or `man iconv_open` should give a clue which encodings are available.

## Use pdfroot

The `pdfroot` variable allows you to store the paths to PDF or Postscript offprints of your references in a more efficient and more flexible way than a full path.

The L1 field of a reference can hold the path to an electronic version of the document. This path will be converted into a link in the HTML output, and the neat plan is to open the document the link points to with a simple mouseclick. If you just store the full path of this document, something like `file:///home/markus/literature/5503.pdf`, you may run into problems fairly soon: If you have to reorganize the folder structure in your directory tree, all paths in the database would be incorrect. If you access the database from a different computer as the one that physically stores your offprints, you will be out of luck with these paths as well.

The idea is to store the invariable part of the path in the reference entry and any variable part of the path in the `pdfroot` variable. The `pdfroot` variable can be set either in your configuration file or with a command line option. Consider e.g. the following situation: You decide for some obscure reason to store the offprints in subdirectories according to the publication year. The full paths will thus be something like `/home/markus/literature/1999/2345.pdf` and `/home/markus/literature/2001/6213.ps`. The partial paths in the L1 field of the reference entry could then be set to `file://1999/2345.pdf` and `file://2001/6213.ps`, whereas the correct value for the `pdfroot` would be `/home/markus/literature/`. Simple concatenation by RefDB will then result in the proper full path.

If you now decide to rearrange your hard drive, all you need to do is to set `pdfroot` properly, as long as you don't change the directory structure below `/literature` (if you do that, you shoot yourself in the foot anyway).

If you want to access your offprints from a different computer, you have several options:

- You can mount the directory tree that actually holds the data. If the computer that stores your offprints e.g. exports `/home` as an NFS share, you could mount that to `/mnt` on your local computer. Then you can set `pdfroot` to `/mnt/markus/literature/` to access your offprints.

- You can store the offprints in a location that is accessible with an internet protocol like HTTP or FTP. In that case it is most practical to just store the full URL in the L1 field, as in `http://some.machine/literature/Miller1999.pdf`.

## Interaction with external applications

RefDB has been designed to be as modular and as lightweight as possible. This means that you need some external applications for various purposes. This chapter briefly discusses how you can simplify your work with these external applications.

### Editor

If you want to modify references in your database or write new entries from scratch, you need some suitable editor. Emacs and vim can directly interact with RefDB using special modes or support files. Any other editor will do as well as long as it creates the correct line endings: RefDB wants a plain Unix-style LF (0x0A), not a DOS-style CR LF (0x0D 0x0A). The prime contenders are vi and Emacs which are available on almost any Unix-style operating system as well as on Windows. For the Windows platform there are also a number of free- and shareware editors which let you select the line endings.

Creating new references is straightforward: Type them in, save them, and use refdbc to upload them. If you want to edit existing references in your database, you download them with the `getref` command to a disk file. Open this file with your editor, edit the contents, and save the file again for uploading. If you want to edit several references in a row, you can simply reuse the filename for downloading, then refresh the buffer in your editor (e.g. C-x C-f RET in Emacs). After saving the changes, simply recall the last upload command in refdbc with the up key and press Enter.

Some editors like Emacs and vim support RefDB by means of add-on packages.

### Viewer

While a pager will do in many cases to view the query results, a web browser is a nice alternative. The RefDB command `getref` can generate HTML output at your request with the `-t html` option. Save the output to a file and view this file with your favourite web browser. When you run the next query, reuse the filename and hit the reload button of your browser to display the new results.

A web browser is also a convenient way to print references.