# Tools for reference and notes management

The following tasks can be performed with the tools described in this chapter:

- Manually add, update, and delete reference entries

- Search for and display reference entries

- Search for and display authors and keywords

- Include existing references into personal reference list

- Convert reference data from various formats to RIS

We'll first introduce the command-line client `refdbc`, RefDB's own multi-purpose reference and notes management client, along with the reference data converters that you can use along with it. The following sections explain the input and output data formats as well as the query language used by RefDB.

## Tools

The first subsection explains the usage of the command-line client refdbc.

Whenever you have an electronic source for reference data, you should use these instead of typing datasets from scratch. Reference data come in a variety of formats. Therefore RefDB ships with a few conversion utilities which are discussed in this chapter. These utilities create tagged RIS data from the input data.

Please visit Chris Putnam's Bibutils project for another set of bibliographic data converters. These tools allow to convert a few additional formats like ISI and MODS to RIS which can then be imported by RefDB.

Since version 0.9.3, RefDB supports risx as an additional native input format. You can employ the standard techniques of SGML and XML transformations, i.e. by running DSSSL or XSLT scripts with a suitable engine, to turn bibliographic data encoded as SGML or XML documents into risx.

Remember that RefDB accepts only four character encodings for XML input data: UTF-8, UTF-16, ISO-8859-1, and US-ASCII. If your input data use a different character encoding, please use the command-line utility `iconv` (usually part of the libiconv package) to convert your data to one of the existing character encodings. Do not forget to specify the character encoding in the processing instructions of the input file, otherwise RefDB will assume the data are encoded as UTF-8.

refdbc.1.xml appears to be missing bib2ris.1.xml appears to be missing db2ris.1.xml appears to be missing en2ris.1.xml appears to be missing db2ris.1.xml appears to be missing db2ris.1.xml appears to be missing refdb-pubmed.1.xml appears to be missing refdb_dos2unix.1.xml appears to be missing refdb_latex2utf8txt.1.xml appears to be missing

## Reference data output formats

refdbd implements several standard backends for data output. You can select them with the `-t` option of the `getref` command in refdbc. Remember that all backends just send data. It is your decision what you want to do with the information. You can either view the data on the screen or pipe them to another application or write them to a disk file.

### scrn

The screen backend provides a basic data output for viewing in a terminal, preferably through a pager. By default, the reference ID, the publication year, the authors, the title, and the source information are displayed. You can use the `-s` option to additionally display the abstract (AB or N2), the notes (N1), the reprint info (RP), the address (AD), the publisher (PB), the city (CY), the ISSN/ISBN (SN), the URL (UR), and the user (U1 through U5) and misc (M1 through M3) fields. In addition, the pseudo-field codes NX (all notes) and LX (all links, including UR and DOI) can be used. To specify several fields, simply concatenate the field codes, as in `-s ADPBRP`. `-s ALL` will display all available fields.

### html

The html backend works just like the scrn backend, but encodes this information in a HTML text. This comes in handy if you would like to view the results of your queries in a web browser rather than in a terminal window. This is also the easiest way to obtain fairly nice-looking printed output of your reference data. You simply use the `-o` switch to write the results of your queries to a file, reusing the same filename for each query. After each query you just have to hit the reload button of your browser to view the results of the most recent query.

The visual appearance of the generated HTML files can be customized using CSS files. The URL of the CSS file must be specified with the refdbc `-G` command-line option or by setting the configuration variable `cssurl`. The global configuration file `/usr/local/etc/refdb/refdbcrc` is preconfigured with a `cssurl` entry pointing to the default CSS file installed in `/usr/local/share/refdb/css`. If you want a different appearance, it might be prudent to create a copy of this CSS file and customize it as you see fit. The following element classes can be customized. In most cases, the class name reflects the database field to be formatted:

- H1.h1

- H2.id

- P.title

- P.authors

- P.abstract

- P.note

- P.address

- P.city

- P.publisher

- P.m1, P.m2, P.m3

- P.u1, P.u2, P.u3, P.u4, P.u5

- P.url

- EM.periodical

- EM.volume

- EM.issue

- EM.page

### xhtml

This is a variant of the html output which creates valid XML output according to the XHTML V1.0 Transitional DTD.

### ris

This is identical to the input format. Use it to export references to other reference management systems.

### risx

Retrieves the data as an XML document using the risx DTD. Use it to edit references or to create backups of your databases. The default output includes a docline and an internal subset declaring some common entities. This format works ok for DTD-based processing systems. If you require namespaced output for further processing with schema-based tools, use the `-n` option to declare a namespace prefix.

### mods

This backend retrieves the data as an XML document using the MODS schema. This is another useful exchange format for bibliographic data. Use the `-n` option to retrieve namespaced output with the given namespace prefix.

### bibtex

This backend provides output formatted for use as a bibtex reference database. This can be used with the tex and bibtex applications to create bibliographies for documents written with Donald Knuth's famous TeX typesetting system. The `-s` option cannot be used with this backend and will be ignored.

### db31

The DocBook SGML backend formats the query result as a bibliography element in a SGML document using the DocBook DTD. RefDB outputs an appropriate doctype string at the beginning of the data. The string is commented out so the contents can be directly inserted into a larger document by some processing application. If you need the data as a standalone document, simply use a script to uncomment the first line. The `-s` option cannot be used with this backend and will be ignored. This option is called db31 to distinguish it from any later additions that may be necessary due to possible incompatibilities in new DocBook versions. It just means that the SGML that this backend creates should work with DocBook V.3.1 or later unless a new backend will be added.

### db31x

The output is essentially the same as with the preceeding backend but you'll get a DocBook XML document instead. The default output works for DTD-based processing with DocBook XML versions 4.x. If you need namespaced output for schema-based processing, use the `-n` option to define a namespace prefix.

### db50x

This backend outputs schema-based DocBook V5. The default output declares a default namespace. You can use the `-n` option to define a namespace prefix which will be used in all elements.

### teix

The TEI XML backend formats the query results as a TEI P4 listBibl element. RefDB outputs an appropriate processing instruction and doctype string at the beginning of the data. The string is commented out so the contents can be directly inserted into a larger document by some processing application. If you need the data as a standalone document, simply use a script to uncomment the first line. The `-s` option cannot be used with this backend and will be ignored.

### tei5x

This backend outputs schema-based TEI P5. The default output declares a default namespace. You can use the `-n` option to define a namespace prefix which will be used in all elements.

## Extended notes output formats

RefDB implements a few backends to output extended notes. You can select them with the `-t` option of the `getnote` command.

### scrn

The screen backend provides the output most suitable for viewing notes in a terminal. Send the output through a pager for best results. Use the `-s NL` or `-s ALL` options to display all objects the note is linked to.

### html

The information returned by this backend is encoded as a HTML document. Use this format to print nicely formatted notes from your web browser. Otherwise the same applies as said for the scrn backend. See above for some hints about formatting the output with a CSS file.

### xhtml

The output is the same as for the html backend but the output is formatted as an XHTML document according to the XHTML V1.0 Transitional DTD.

### xnote

This backend encodes the output in xnote documents, the same as the input format for extended notes.

## The query language

The `getref` command is probably the most heavily used command. You use it to retrieve the references that you collected and saved in the database. To find a certain article or several related articles, all you have to do is to express your query in a language that RefDB understands. The first section describes how to formulate search strings for your queries.

The `getnote` command used to locate extended notes is very similar. The specifics of this command will be described in the subsequent section.

### The reference query language

The syntax for the search string follows these rules:

- You can search for any fields in the RefDB database. As a matter of fact, you have to specify at least one field for your query. Something like `getref *` will not work, but you may use `getref ":ID:>0"` instead to list all entries in the database (beware, this may be a lot).

- Every search item has the following general form:

  :XY:[=|~|!=|!~|<>|<|>|<=|>=]string

  The current implementation of RefDB is very picky about spaces. Please make sure that you do not insert spaces or other whitespace on either side of the operators ("=", "~" and so on). If your value happens to start with a space, include the value in quotation marks or protect the space with a backslash.

  The sequence ":XY:" denotes the reference data field to search in. The names are mostly taken from the RIS specification. Possible field names are:

  :TY:

    Type of the reference.

  :ID:

    The unique identifier of a reference. This is the numeric identifier that RefDB assigns to each new reference.

  :CK:

    The unique citation key of a reference. This is the alphanumeric string that was either supplied by the user or automatically generated by RefDB.

  :TI:, :T2:, :T3:, :TX:

    The title of the reference, of the secondary title, and of the series title, respectively. :TX: performs a search in all title levels.

  :AU:, :A2:, :A3:, :AX:

    The name of an author, of a secondary author/editor, and of a series author, respectively. :AX: performs a search in all author levels.

  :PY:, :Y2:

    The publication date and the secondary date, respectively.

  :N1:

    The notes that user can add to the reference.

  :KW:

    A keyword.

  :RP:

    The reprint status of the reference.

  :AV:

    The location of an offprint (physical, URL, or path)

  :SP:

    The start page.

  :EP:

    The end page.

  :JO:, :JF:, :J1:, :J2:

    The abbreviated name, the full name, the user abbreviation 1, and the user abbreviation 2 of a journal name, respectively.

  :VL:

    The volume number.

  :ED:

    The name of an editor.

  :IS:

    The issue (article) or chapter (book part) number.

  :CY:

    City of publication of a book.

  :PB:

    Name of the publishing company.

  :U1: through :U5:

    The user-defined fields 1 through 5. In certain cases, U5 contains the number of the paper copy of the reference (see the `addref` command).

  :N2:

    The abstract of the reference.

  :SN:

    The ISSN or ISBN number.

  :L1: through :L4:

    The link fields 1 through 4.

  :AD:

    The address of the contact person.

  :UR:

    The URL of a web page related to the reference.

  Some pseudo-fields help to retrieve the contents of the RIS fields M1 through M3. RefDB provides these pseudo-fields to better identify the contents of these fields. Please be aware that these pseudo-fields apply only to particular reference types. If in doubt, consult the Reference Manager manual which contains a list of how the M1-M3 fields are used by each reference type.

  :DO:

    The digital objects identifier.

  :TO:

    The type of work.

  :AR:

    The area of a map

  :OS:

    The operating system of a computer program

  :DG:

    The type of degree of a thesis.

  :RT:

    Running time of audiovisual data.

  :CI:

    International class code of a patent.

  :CU:

    US class code of a patent.

  :SE:

    The email address of the sender.

  :RE:

    The email address of the recipient.

  :MT:

    The type of media.

  :NV:

    The total number of volumes in a series

  :EI:

    The edition of a book.

  :CO:

    The type of computer used for a software or data.

  :CF:

    The location of a conference

  :RN:

    The registry number.

  :CL:

    The classification of an audiovisual material.

  :SC:

    The section of a magazine or newspaper

  :PN:

    The number of a pamphlet.

  :CN:

    The number of a chapter.

  In addition to the above field specifiers, there are a few that allow to retrieve references based on extended notes attached to them:

  :NID:

    The ID of an extended note.

  :NCK:

    The alphanumeric key of an extended note.

  References are matched according to these rules:

  - The alphanumerical fields are matched by (non-)equality to a literal string or to a regular expression. For literal matches the operators "=" and "!=" are accepted, denoting equality and non-equality, respectively. The search-string is a plain-text string.

    Along the same lines, "~" and "!~" denote equality and non-equality for regular expression matches. The search-string can contain any legal characters and constructs as in standard Unix regular expressions. By default, the query matches if the search string is contained anywhere in the target string. If you need a left-match, a right-match, or a full match, use the regexp special characters "^" (match the beginning of a line) and "$" (match the end of a line) to your needs. For further information about regular expressions, see the section regular expressions

    Some database engines, like SQLite, do not support Unix-style regular expressions. Use SQL regular expressions instead.

    If you use regular expressions, be aware that you will have to escape characters with a special meaning if you want them to be matched literally. For further details, see the examples below.

  - The numerical fields can be matched by equality, non-equality, or by greater-than/less-than comparisons. In these cases, use "!=", ">", and "<" instead of the "=", respectively.

  - If the search-string contains spaces, the whole string must be enclosed by single quotation marks or the spaces must be escaped with a backslash "\".

- Several search items can be combined by the Boolean operators "AND", "OR", " AND NOT". They can be grouped by brackets "()".

  If you use the operator "AND NOT", the search item it refers to has to be enclosed in brackets. See the examples below.

- The author (:AU:) and keyword (:KW:) searches allow an abbreviated syntax if you search for several authors or keywords at a time. The construct:

  :AU:=name1 AND :AU:=name2 AND :AU:=name3 ...

  can be written as:

  :AU:='& name1 name2 name3 ...'

  To specify an "OR" search, use "|" instead of "&".

  If an item in the search contains spaces, they have to be escaped by backslashes, as in "Amino\ Acid\ Sequence".

### The notes query language

The getnote command uses essentially the same query language as described above. However, there is a different set of field specifiers that you can use to locate extended notes:

:NID:

  The unique id of an extended note.

:NCK:

  The unique citation key of an extended note.

:NPY:

  The date of an extended note.

:NTI:

  The title of an extended note.

:NKW:

  A keyword. This is a keyword attached to a note in order to categorize the latter, similar to a keyword in a reference.

:KW:

  A keyword. This is a keyword that the note is linked to, i.e. a keyword that the note was attached to in order to supply additional information.

:AU:

  The name of an author or editor. Use this field specifier to locate notes that are linked to a particular author.

:JF:, :JO:, :J1:, :J2:

  The full, abbreviated, or user-abbreviated name of a periodical. Use this field specifier to locate notes that are linked to a periodical.

:ID:

  The id of a reference. Use this field specifier to locate notes linked to a particular reference.

:CK:

  The citation key of a reference. Use this field specifier to locate notes linked to a particular reference.

### Some example queries

This section shows a few example queries to help you get familiar with the syntax. If you are not familiar with the regular expressions used here, please peruse the regular expressions section. We will not use any of the fancy switches of the `getref` command here, so the output will always be a simple listing on the screen.

These examples assume that your database engine supports Unix regular expressions. This holds true for MySQL and PostgreSQL, wherease SQLite uses the simpler SQL regular expressions instead.

We'll start with some easy queries. First we want to display a reference with a specific ID (25 in this example):

    	  refdbc:
    	  getref :ID:=25

Next we want to list all references by a specific author. We'll use only the last name here. If several authors share this last name, we have to specify the initials as well, as shown in the second example. In the first example we use a regular expression match, denoted by the tilde operator. This obviates the need to know the full name precisely. The second example uses a literal match instead. Note the use of the caret "^" in the first example which makes sure that the name actually starts with the capital M. Otherwise, a last name like "DeMillerette" would match as well. This trick is not required in the second example as the literal match always implies a full match.

    	  refdbc:
    	  getref :AU:~^Miller

    	  refdbc:
    	  getref :AU:=Miller,J.D.

If Dr. Miller was a productive person, our previous query may have returned dozens of references. Now we try to filter out the paper or the papers that we really need. In the next example, we restrict the results to the years 1995 through 1999:

    	  refdbc:
    	  getref :AU:~^Miller AND :PY:>1994 AND :PY:<2000

If this did not bring us close enough, we may try to include a coauthor:

    	  refdbc:
    	  getref :AU:=~Miller AND :AU:~^Doe AND :PY:>1994 AND :PY:<2000

At this point we could narrow down the search by excluding other authors that often published with Dr. Miller, but are irrelevant here:

    	  refdbc:
    	  getref :AU:~^Miller AND :AU:~^Doe AND NOT (:AU:~^Jones) AND :PY:>1994 AND :PY:<2000

Unfortunately, this is still a venerable list of publications. Now we try to include a few keywords. This is now a pretty complex query. It will return all references by the authors Miller and Doe between 1995 and 1999 with either the keyword "blood" or the keyword "animal" or the keywords "guanyl" and "cyclase", the latter only if both are present. The truncated spelling of "guanyl" ensures that both "guanylyl" and "guanylate" (which are interchangeable) will match. The funny expressions with the angle brackets ensure that the keywords will match regardless of whether they start with a capital letter or not.

    	  refdbc:
    	  getref :AU:~^Miller AND :AU:~^Doe AND :PY:>1994 AND :PY:<2000 AND
    	  (:KW:~[bB]lood OR :KW:~[aA]nimal OR (:KW:~[gG]uanyl AND :KW:~[cC]yclase))

And now for something completely different. If you've added a couple extended notes to your database, you can retrieve references that are attached to a specific extended note, e.g. to the note with the citation key "biochemistry1999":

    	  refdbc:
    	  getref :NCK:=biochemistry1999

If you want to see all notes which are attached to a reference with the citation key "Miller1999", use the following command:

    	  refdbc:
    	  getnote :CK:=Miller1999

Regular expressions may have unwanted side effects at times. Consider the keyword "52-67-5 (Penicillamine)" (a chemical name as used by the Pubmed database). Doing a literal match is straightforward:

    	  refdbc:
    	  getref :KW:='52-67-5 (Penicillamine)'

However, if we use the same argument for a regexp match, we won't get the desired results. The parentheses have a special meaning in regular expressions. Therefore we have to escape them if we want a literal match:

    	  refdbc:
    	  getref :KW:~'\(Penicillamine\)'

Things are a little different again if you run a database engine that does not use Unix regular expressions, but SQL regular expressions instead. These know only '%' and '_' as special characters, and you have to escape them by doubling:

    	  refdbc:
    	  getref :KW:~'100%%'

Remember that if you extend or modify a previous query, you don't have to retype everything: Just use the up arrow key to scroll through the previous commands, or use Ctrl r to search for a specific query in the history.

## Regular expressions

This section provides a brief overview over regular expressions. In the context of RefDB, we have to deal with two flavors of regular expressions: Unix-style and SQL. The former are more important as we use them to write queries. The latter are used sparingly, e.g. to search the filenames of databases.

Some database engines like SQLite do not support Unix-style regular expressions. You have to use SQL regular expressions in this case.

The difference between a literal match and a regular expression match is that the latter allows some fuzziness in the search string. The former requires that the search string and the search result match character by character. In simple words, regular expressions allow to search for strings which are similar to some extent, and you can exactly specify to which extent.

### Unix-style regular expressions

Regular expressions distinguish between regular characters and special characters (meta characters). The simplest regular expressions actually don't look like regular expressions, as the following example shows:

    foo

This will search for the string "foo" at any position in the target elements. This would find strings like foobar, lifoo, or lifoobar. That is: if there are no meta characters, a simple string match is attempted, however at any position in the element. This is different from search strategies in some other databases where a full match or a left-match is attempted by default.

We can now replace one o in the above sample with a meta character. We use the . (dot) which matches any single character, including a newline, at that position:

    f.o

This will find strings like fao, fdo, but as well all strings of the previous example.

Another very common meta character is the *, which matches zero or more instances of the previous character. Thus,

    fo*

will now find things like fo, foooo, but also fbar and lifooobar. The meta character + is similar, but requires at least one instance of the previous character:

    fo+

This would retrieve all strings of the last example except fbar as this contains the o zero times.

    fo?

The questionmark meta character will retrieve either zero or one instances of the previous character. This would match f and fo, but not foo.

The meta characters ^ and $ are important to determine the relation of the search string to the line start or line end:

    ^foo

This will match foo only if it is located at the line start. Similarly,

    foo$

will find foo only when it is located at the line end. If you combine these two like in the next example:

    ^foo$

foo will be found only if this is the complete element, starting and ending the line.

The following list briefly explains some more terms which are helpful in regular expressions.

()

  Use the round brackets to group characters to a sequence. This is particularly useful with the above mentioned metacharacters *, +, and ?.

      (foo)*

  This will match zero or more instances of the sequence foo. It will find e.g. foo and foofoo, but not fofo.

[]

  matches any single character between the brackets.

      [0-9]

  This will match any digit. Continuous ranges of characters can be indicated with a dash, as seen here.

[^]

  matches any single character except the ones between the brackets

      [^abc]

  This will match any character except a, b, and c.

\

  The backslash escapes the following meta character and treats it as a literal character.

      \.

  This will match only the dot instead of any single character.

\{n,m\}

  This will find n to m repeats of the previous character.

      fo\{2,3\}

  This regular expression will find foo and fooo, but not fo or foooo.

For further information about regular expressions, see the regex chapter in the MySQL documentation.

### SQL regular expressions

SQL regular expressions are much simpler, as there are only two metacharacters:

%

  matches any string

_ (underscore)

  matches any single character

In order to match a SQL regular expression special character literally, you have to escape it by doubling.