# Quickstep Hartree Fock eXchange - LiH (Regression Tests)

Hybrid benchmark to test CP2K scaling up to 10000s of cores

## Description

This is a single-point DFT energy calculation using Quickstep GAPW (Gaussian and Augmented
Plane-Waves) with hybrid Hartree-Fock exchange. It consists of a 216 atom Lithium Hydride crystal
with 432 electrons in a 12.3 cubic angstrom cell. These types of calculations are generally around
one hundred times the computational cost of a standard local DFT calculation, although this can be
reduced using the Auxiliary Density Matrix Method (ADMM). Using OpenMP is of particular benefit here
as the HFX implementation requires a large amount of memory to store partial integrals. By using
several threads, fewer MPI processes share the available memory on the node and thus enough memory
is available to avoid recomputing any integrals on-the-fly, improving performance.

## Files description

- [`input_bulk_B88_3.inp`](input_bulk_B88_3.inp): needed to generate an initial wfn (wave function)
  file for the HFX runs (this should be run once before running the actual HFX benchmark, and is not
  a part of the benchmark)
- [`input_bulk_HFX_3.inp`](input_bulk_HFX_3.inp): the actual input file for the HFX benchmark
- the additional files [`t_c_g.dat`](../../data/t_c_g.dat) and [`POTENTIAL`](../../data/POTENTIAL)
  are needed, and can be found in the `cp2k/data/` directory.

## Benchmark Requirements

To run these this benchmark, CP2K needs to be compiled with libint support (-D\_\_LIBINT), and it is
advantageous to have a OMP/MPI hybrid code (cp2k.psmp).

## How to Run the Benchmark

1. as a preliminary step, not part of the benchmark: run `input_bulk_B88_3.inp` (5min. with 256x1
   mpixomp tasks) and rename the resulting wavefunction file `LiH_bulk_3-RESTART.wfn` to `B88.wfn`

   ```shell
   cp LiH_bulk_3-RESTART.wfn B88.wfn
   ```

1. run `input_bulk_HFX_3.inp` on a number of nodes (about 30min on 2048 cores)

   The obtained energies should be similar to (obtained on 2048 MPI x 8 OMP cores):

   ```cp2k-output
   1  OT CG          0.15E+00    139.386     0.0014482195     -870.3838179520
   2  OT LS          0.30E+00     19.767     1.0000000000     -870.7951031236
   3  OT CG          0.30E+00     20.724     0.0001614074     -870.9275954835
   4  OT LS          0.31E+00     19.665     1.0000000000     -870.9346994255
   5  OT CG          0.31E+00     20.641     0.0000168390     -870.9347195049
   6  OT LS          0.29E+00     19.760     1.0000000000     -870.9347906736
   7  OT CG          0.29E+00     21.192     0.0000009576     -870.9347911831
   ```

## Notes

The amount of memory available per MPI process must be altered according to the number of MPI
processes being used. If this is not done the benchmark will crash with an out of memory (OOM)
error. The input file keyword `MAX_MEMORY` in `input_bulk_HFX_3.inp` needs to be changed as follows:

```cp2k-input
MAX_MEMORY 14000
```

should be changed to

```cp2k-input
MAX_MEMORY new_value
```

The new value of `MAX_MEMORY` is chosen by dividing the total amount of memory available on a node
by the number of MPI processes being used per node. If a shorter runtime is desirable, the following
line in `input_bulk_HFX_3.inp`:

```cp2k-input
MAX_SCF 20
```

may be changed to

```cp2k-input
MAX_SCF 1
```

in order to reduce the maximum number of SCF cycles and hence the execution time. If the runtime or
required memory needs to be reduced so the benchmark can run on a smaller number of nodes, the OPT1
basis set can be used instead of the default OPT2. To this end, the line

```cp2k-input
BASIS_SET OPT2
```

in `input_bulk_B88_3.inp` and in `input_bulk_HFX_3.inp` should be changed to

```cp2k-input
BASIS_SET OPT1
```

## Results

### Best Configurations

The best configurations are shown below. Click the links under "Detailed Results" to see more
detail.

| Machine Name | Architecture |       Date | SVN Revision | Fastest time (s) | Number of cores |                  Number of threads |                                                        Detailed results |
| -----------: | -----------: | ---------: | -----------: | ---------------: | --------------: | ---------------------------------: | ----------------------------------------------------------------------: |
|       HECToR |     Cray XE6 |  21/1/2014 |    13196(\*) |          121.362 |           65536 |         8 OMP threads per MPI task |       [hector-lih-hfx](https://www.cp2k.org/performance:hector-lih-hfx) |
|       ARCHER |    Cray XC30 |   9/1/2014 |    13473(\*) |           51.172 |           49152 |         6 OMP threads per MPI task |       [archer-lih-hfx](https://www.cp2k.org/performance:archer-lih-hfx) |
|       Magnus |    Cray XC40 | 10/11/2014 |    14377(\*) |           62.075 |           24576 |         4 OMP threads per MPI task |       [magnus-lih-hfx](https://www.cp2k.org/performance:magnus-lih-hfx) |
|    Piz Daint |    Cray XC30 | 12/05/2015 |        15268 |           66.051 |           32768 | 4 OMP threads per MPI task, no GPU | [piz-daint-lih-hfx](https://www.cp2k.org/performance:piz-daint-lih-hfx) |
|       Cirrus |   SGI ICE XA | 24/11/2016 |        17566 |          483.676 |            2016 |         6 OMP threads per MPI task |       [cirrus-lih-hfx](https://www.cp2k.org/performance:cirrus-lih-hfx) |
|       Noctua |   Cray CS500 | 25/09/2019 |      9f58d81 |          131.290 |           10240 |         4 OMP threads per MPI task |       [noctua-lih-hfx](https://www.cp2k.org/performance:noctua-lih-hfx) |

(\*) Prior to r14945, a bug resulted in an underestimation of the number of ERIs which should be
computed (by roughly 50% for this benchmark. Therefore these results cannot be compared directly
with later ones.

### 2009-04-15

Running on ORNL's Cray XT5 (Jaguar) the following runtime has been obtained in a setup using 8 OMP
threads per node (8 cores per node / 16 Gb per node).

| Cores | Full CP2K\[s\] | HFX\[s\] | local HFX\[s\] | Mem/node\[Mb\] |
| ----: | -------------: | -------: | -------------: | -------------: |
|  1024 |        6569.07 |  6488.24 |        6406.99 |       14845.00 |
|  2048 |        1369.92 |  1314.86 |        1259.82 |       13468.00 |
|  4096 |         722.24 |   670.18 |         631.11 |        7293.00 |
|  8192 |         402.18 |   352.21 |         317.59 |        4306.00 |
| 16384 |         274.41 |   213.88 |         172.88 |        2801.00 |
| 32768 |         201.71 |   135.47 |          85.29 |        2046.00 |
| 65536 |         255.97 |   117.33 |          41.43 |        1673.00 |
