This document gives information on mixed precision development in hypre.

## General information

Terminology:
* Multiprecision - a solver, function, or feature that uses different precisions individually.
* Mixed precision - a solver, function, or feature that uses different precisions simultaneously.

The mixed precision support in hypre is primarily intended to:
* provide multiple precisions at runtime, and
* provide faster and more memory efficient mixed-precision solvers.

Features of mixed precision library configuration include:
* Existing user code runs as before without requiring modification.
* All compile-time precisions are available at runtime.
* Precision can be set globally at runtime and requires only minimal changes to existing user code (to manage real-valued data).
* Precision can be assigned to objects (e.g., matrices) which in turn determines the precision of related methods (e.g., Matvec). [Future feature]
* Additional mixed precision solvers are available.

### Basic structure

Each directory in hypre contains a number of files devoted to multiprecision and mixed precision.  Most of the files either begin with `mup` or have `mup` in their names.  Since header file names should begin with either `HYPRE_` or `_hypre_`, the header files associated with multiprecision have names such as `HYPRE_utilities_mup.h` or `_hypre_utilities_mup.h`.  Most of the multiprecision files in each directory are automatically generated to keep code development and maintenance to a minimum.

With only a few exceptions, every function `Foo` in hypre is provided in multiple precisions.  This is achieved by compiling the library three times to generate the symbols `Foo_flt`, `Foo_dbl`, and `Foo_long_dbl`, plus a smaller fourth compile of auto-generated wrapper implementations of `Foo`.  These wrapper versions of `Foo` either default to a fixed precision determined at compile time or branch to a precision determined at runtime.

Each function must be listed in one of three files:
- `mup.fixed` - `Foo` has fixed compile-time precision (configured by the user)
- `mup.functions` - `Foo` has precision determined by the global runtime precision
- `mup.methods` - `Foo` has precision determined by its object(s) precision(s) [Future feature]

There is an optional file called `mup.exclude` that can be used for functions that should only be compiled once.  The only use of this currently is in `utilities`, and it requires special treatment at build time for the files that implement the listed functions.

The following table shows the symbols generated by each of the four compiles for a function `Foo` in one of the above files (the `MP_BUILD_*` variables are used in both the autotools and CMake builds to indicate the precision of the compile).

| Build Type / Function List | mup.fixed     | mup.functions | mup.methods   |
|----------------------------|---------------|---------------|---------------|
| MP_BUILD_SINGLE            | Foo_flt       | Foo_flt       | Foo_flt       |
| MP_BUILD_DOUBLE            | Foo_dbl       | Foo_dbl       | Foo_dbl       |
| MP_BUILD_LONGDOUBLE        | Foo_long_dbl  | Foo_long_dbl  | Foo_long_dbl  |
| No MP (standard compile)   | Foo           | Foo, Foo_pre  | Foo, Foo_pre  |

For `mup.functions` and `mup.methods`, the `Foo` function calls `Foo_pre` together with a precision argument.  Developers may call `Foo_pre` directly when writing mixed precision code.

## Workflow for developers

A high-level summary of the developer workflow is as follows, with additional details given below:
- Implement code as before.
- Run `mup_check` from the `src` directory and update function lists.
- Run `mup_code` from the `src` directory to generate multiprecision code.
- Compile and debug with mixed precision configured.
- Run `mup_clean` from the `src` directory to remove miscellaneous files.
- Commit code, including updated and newly generated multiprecision code.

### Updating function lists

Developers should implement, compile, and debug code as before.  If any new functions are implemented, developers should run the following script from the `src` directory:
```
mup_check
```
This script will compile the code, extract all function names from the library file, and compare it to the existing `mup.*` function lists.  If functions need to be adjusted in these lists, the script will indicate that an `UPDATE` is needed.  Otherwise, it will indicate `OK`.  Use the appropriate `mup_check.err` file to see what needs to be added or deleted.

Functions should appear in only one of the `mup.*` function lists.  The current guidance is to put all external user API functions (e.g., `HYPRE_Foo`) in either `mup.functions` or `mup.methods` [future feature], and put all other functions in `mup.fixed`.  When writing mixed-precision code, developers may move functions from `mup.fixed` to either `mup.functions` or `mup.methods` [future feature] as needed (e.g., to make use of internal `hypre_Foo_pre` routines in mixed-precision AMG).

### Generating new multiprecision code

If any function lists are updated or any prototypes changed, it will be necessary to run the following script from the `src` directory:
```
mup_code
```
This top-level script calls individual `mup_code` scripts in each subdirectory to generate code in the files `mup_*.c` and to generate header information (e.g., prototypes and macros) in the files `HYPRE_*mup.h` and `_hypre_*mup*.h`.  The individual `mup_code` scripts call additional scripts in the `src/config` directory, and they are tailored to each individual directory.  In particular, some directories such as `ParaSails` do not require any wrapper code generation because user interfaces are provided by way of other directories such as `parcsr_ls`.  Here, only certain header files are auto-generated.

