\documentclass[letterpaper]{book}
\usepackage{amssymb,amsmath,latexsym,hyperref,graphics,color}

\setlength{\parindent}{0pt}
\setlength{\parskip}{2ex plus 0.5ex minus 0.2ex}

\begin{document}

% Title Page
\title{MADNESS Molecular electronic structure calculations}
\date{Last Modification: 7/7/2016}
\maketitle

% Copyright Page
\pagestyle{empty}
\null\vfill
\noindent
This file is part of MADNESS.


Copyright (C) 2007, 2010 Oak Ridge National Laboratory

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public
License as published by the Free Software Foundation; either version 2 of the License, or(at your option) any later
version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free
Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

For more information please contact:
\begin{quote}							
Robert J. Harrison 				\\
Oak Ridge National Laboratory 	\\
One Bethel Valley Road 			\\
P.O. Box 2008, MS-6367			\\
Oak Ridge, TN 37831				\\
								\\
email: harrisonrj@ornl.gov 		\\
tel: 865-241-3937				\\
fax: 865-572-0680	
\end{quote}		
\newpage


% Table of Contents Pages
\clearpage
\setcounter{page}{1}
\pagenumbering{roman}

\setcounter{tocdepth}{10}
\renewcommand\contentsname{Table of Contents}
\tableofcontents


\clearpage
\setcounter{page}{1}
\pagenumbering{arabic}

\chapter{Overview}

\section{Capabilities}



\section{Current status}

The software is still in what might best described as pre-production
quality.  Most things mostly work, but there are enough quirks that
unfortunately it is far from being a black box like most Gaussian
codes.

Using the code to do actual science is the most effective mechanism
for us to find and fix issues, as well as to prioritize implementation
effort.  Please report problems, issues and functionality requests to the
MADNESS issue tracker at XXXXXXXXXXXXXXX.

\chapter{Configuring and building}

This is just for LI-red for the purposes of the summer school.
MADNESS should build out of the box on most Linux and Apple Macintosh
boxes. Intel MKL is now free so you should alway use that (download
from Intel), and there is an open source version of Intel TBB (Linux
distributions should have that in their package managers).  If you are
building to run on a virtual machine you should configure with the
never spin option.

\begin{enumerate}
\item Copy the modules from section \ref{sec:pbslired} below into your
  \verb+.bashrc+ file.  You will be using the Intel compiler, MPI, MKL
  and TBB which is the recommended stack when running on a cluster.

\item Clone MADNESS from
  \url{https://github.com/m-a-d-n-e-s-s/madness}.  I (RJH) used
  revision {\tt a1b4bae8acf57ae363f24cb8a7ea9e7229a0d95f} for testing.

\item {\tt cd madness}

\item \verb+sh autogen.sh+ --- only need to do this after initial checkout

\item \verb+autoreconf+ --- only need to do this if a configure script has changed

\item Configure with this command
\begin{verbatim}
  ./configure --disable-shared \
       CC=mpiicc CXX=mpiicpc +\verb+ MPICC=mpiicc MPICXX=mpiicpc
\end{verbatim}
If you have installed LIBXC you can specify it with option
\begin{verbatim}
  --with-libxc=full-path-to-libxc+.
\end{verbatim}

\item \verb+make -j 10 libraries+ --- using just 10 processes on
  LI-red is to be social.  Also, if building on your own machine note
  that some of the files can use over 2GB of memory to compile, so if
  you have limited memory you may need to restrict the number of
  processes.  If compiling on {\tt cn-mem} I always use
  \verb+make -j libraries+ which uses maximum parallelism (since {\tt
    cn-mem} has 72 cores and 3 TB of memory.

\item \verb+cd src/apps/moldft+

\item \verb+make moldft+

\item \verb+MAD_NUM_THREADS=10 ./moldft+ --- run a quick test (the {\tt input} file by default does LDA water which is quick).

\end{enumerate}


\chapter{Execution}

To run {\tt moldft} with a single, multi-threaded process simply
invoke the name of the executable with the name of the input file on
the command line.  If the input file is ommited, it defaults to a file
named {\tt input}.  E.g.,

\begin{verbatim}
    ./moldft inputfilename
\end{verbatim}


Recommendations:
\begin{itemize}
\item Use Intel TBB if it is available since it is more scalable than the old task pool.
\item By default, MADNESS will create one thread for every core in the
  computer, but even with TBB it may be more effective to use fewer threads
  (see the environment variable \verb+MAD_NUM_THREADS+). E.g.,
\begin{verbatim}
      MAD_NUM_THREADS=10 ./moldft inputfile
\end{verbatim}
\item Oversubscribing the cores can lead to very poor performance due to
  cache contention and locks/mutexes being held by de-scheduled
  threads. Oversubscription can happen due to other processes (of yourself or other users)
  running at the same time, or from use of hyperthreading (which on Intel processors
  makes one phyiscal core appear to the system as two cores).
\item Many threads seem to interact badly with how we are using MPI.
  Thus, it can help both robustness and performance to use multiple
  MPI processes per node with fewer threads per processes. As a side benefit you
  can bind the threads associated with each process to execute on the same
  socket, which improves thread scaling.  See the SeaWulf batch script below
  for an example.
\end{itemize}

To run in parallel using MPI to create multiple, multi-threaded
processes use the command {\tt mpirun} (or on some systems {\tt
  mpiexec}) --- this is very system dependent so you may need
additional options to specify where and how many processes are created
(see below for use on SeaWulf and LI-red).  E.g.,

\begin{verbatim}
    export MAD_NUM_THREADS=10
    mpirun -np 12 ./moldft inputfilename
\end{verbatim}

\subsection{Environment variables}

\verb+MAD_NUM_THREADS+ --- Sets the total number of threads to be used
by MADNESS.  When running with just one process all threads are
devoted to computation (1 main thread with the remainder in the thread
pool).  When running with multiple MPI processes, one of the threads
is devoted to communication.  If you have 8 or more cores per node, it
is recommended to leave at least one free for use by the O/S and MPI.

\verb+MRA_DATA_DIR+ --- Full path to the directory containing the MRA
data files (twoscale and autocorrelation coefficients; Gauss-Legendre
quadrature weights).  This is usually not needed unless you have moved
the build/installation directory.

\verb+MAD_BUFFER_SIZE+ --- Sets the buffer size (in bytes) used by the
active messages (default is 1.5MBytes).  Never needed by moldft?

\verb+MAD_RECV_BUFFERS+ --- Sets the number of receive buffers used by
the communication thread (default is 128 and a minimum of 32 is
enforced).  If you are experiencing hangs when running with MPI,
making this number a bit larger (e.g., 256) can sometimes help. Too
many buffers can cause performance problems.

\verb+MAD_SEND_BUFFERS+ --- Sets the number of outstanding
asynchronous send buffers (default 128 and a minimum of 32 is
enforced).  If you are experiencing hangs when running with MPI try
making this smaller (to throttle the volume communication).

\verb+MAD_NSSEND+ --- Every \verb+MAD_NSSEND+ messages, MADNESS
requests a receipt acknowledgment from the receiver in an attempt to
throttle the volume of communication.  The default is the number of
send buffers.  Set to the value \verb+1+ to force receipt every
message to be acknowledged.  This makes things a bit slower but can
workaround MPI hangs.

\subsection{PBS script for LI-red}
\label{sec:pbslired}

This example makes a 2 hour job running in the short queue on 8 nodes
with one process per node and job name {\tt moldft}.  Each process
will use 20 threads.

\begin{verbatim}
#!/bin/bash                                                                     
#PBS -l nodes=8:ppn=1,walltime=02:00:00                                         
#PBS -q short                                                                   
#PBS -N moldft                                                                  

export EXE=/home/rjh/madness/src/apps/moldft/moldft
export INPUT=benzenehf.in
export OUTPUT=benzenehf.out

export I_MPI_FABRICS=shm:ofa
export MAD_NUM_THREADS=20

cd $PBS_O_WORKDIR

mpirun -ppn 1 $EXE $INPUT >& $OUTPUT
\end{verbatim}

The modules I (RJH) used for testing were (put the following in your \verb+.bashrc+)
\begin{verbatim}
module load shared                                                              
module load torque/5.1.0                                                        
module load maui/3.3.1                                                          
module load gcc/4.9.2                                                           
module load intel/compiler/64/16.0.2/2016.2.181                                 
module load intel/mkl/64/11.3.2/2016.2.181
module load intel-mpi/64/5.1.2/5.1.2.150.XXX
module load intel/tbb/64/4.4.3/2016.2.181
module load intel/vtune/2016/2016.1.1.434111
\end{verbatim}

\subsection{PBS script for SeaWulf}

On SeaWulf the PBS option {\tt ppn} is presently not consistent with
that on LI-red --- this will change soon.  In the meantime the script
is a little more complicated in order to get just two MADNESS
processes per node with threads able to use all cores.

ASIDE: SeaWulf has 28 cores per node and a 40 Gbit/s InfiniBand
network whereas LI-red has 24 cores per node and a 56 Gbit/s network.
Somehow these factors (or others?) combine to make MADNESS execution
on SeaWulf with 1 MPI process that uses 26 MADNESS threads/node not
reliable.  Hence, the job below uses two MPI processes per node each
with 11 MADNESS threads.

\begin{verbatim}
#!/bin/bash
#PBS -l nodes=12:ppn=28,walltime=10:00:00
#PBS -q default
#PBS -N geomopt

# Probably best to put these in your .bashrc so don't need here
module load intel/compiler/64/16.0/2016.1.056.XXX
module load intel/mkl/64/11.3/2016.1.056.XXX
module load intel-mpi/64/5.1.2/5.1.2.150.XXX
module load intel/tbb/64/4.4.3/2016.2.181
export INTEL_LICENSE_FILE=28518@129.49.83.234

# Intel MPI should use shared-memory and OFED for communication
export I_MPI_FABRICS=shm:ofa

# We will create two processes per node --- bind to separate sockets
export I_MPI_PIN_DOMAIN=socket

# Should be set elsewhere but may not be
export I_MPI_HYDRA_BOOTSTRAP=rsh

# 14 cores per socket = Linux+MPI+unknown+madness
export MAD_NUM_THREADS=11

# Change to the directory from which the job was submitted
cd $PBS_O_WORKDIR

# Presently SeaWulf is configured so that to get a dedicated node we
# need to set ppn=28 above which gives us 28 processes per node, but we
# want two.  So need to make our own host file.
sort < $PBS_NODEFILE | uniq > $$.hosts
let NPROC=$PBS_NUM_NODES*2

# Paths to files
MOLDFT=/gpfs/home/rharrison/madness/src/apps/moldft/moldft
INPUTFILE=input
OUTPUTFILE=output

# Redirect output to a file so that can see it while job is running

mpirun -f $$.hosts -ppn 2 -n $NPROC $MOLDFT $INPUTFILE  >& $OUTPUTFILE
  
\end{verbatim}


\section{Input structure}

Presently the input {\em must} contain both a \verb+dft+ and a
\verb+geometry+ input block.  The file is scanned for the first such
block (so you can have multiple blocks in a file --- just the first is
used). Within these blocks, lines beginning with \verb+#+ are read as comments.
Outside these blocks, data is ignored.

\section{Reproducibility}

For small molecules, the calculations should usually be fully
reprodicible from run to run or with different numbers of threads or
processes.  However, for larger molecules, the different rounding
error arising from different order of parallel execution can cause
different execution paths.  So results may vary in the last digit or
so, and sometimes more especially if additional/fewer iterations are
made.  However, results at convergence should {\em always} agree to
the that expected from the truncation and convergence thresholds, and
the number of iterations between runs should only differ by about one.
If you are seeing worse behavior than this, then please send the input
and output to the MADNESS issue tracker.

\chapter{Geometry specification}

\begin{verbatim}
  geometry
      units atomic or angstrom (default atomic)
      eprec precision (default 1e-4)
      tag x y z
      pstag x y z charge
  end
\end{verbatim}

The units and energy precision ({\tt eprec}) must presently be specified {\em before} coordinates of atoms are specified.

{\tt units} --- E.g., {\tt units angstrom} --- The user input units which can be {\tt atomic} or {\tt angstrom} (default is {\tt atomic}).  Beware --- presently, the output file always contains atomic units.

{\tt eprec} --- E.g., {\tt eprec 1e-5} --- The energy error per atom
in atomic units due to smoothing the nuclear potential (default is
{\tt 1e-4}).  The default is adequate for most chemical purposes
(giving energy differences to at least {\tt 1e-5} atomic units per atom
and geometries accurate to about {\tt 1e-3} atomic units) unless you
are doing very high accuracy benchmark studies.  Gradients may get
noisy if you make {\tt eprec} too small, though we don't have much
experience with this yet. 

{\tt tag x y z} --- E.g., {\tt Be 0.3 -0.1 3.1} --- Specifies an atom using its atomic symbol (case insensitive) and Cartesian coordinates.

{\tt bqtag x y z charge} --- E.g., {\tt Bq9 20.0 20.0 20.0 -10.0} --- The tag must begin with {\tt Bq} (case insensitive).  It is used to place a charge at an arbitrary location (e.g., to simulate an external field).

{\tt pstag x y z charge} --- E.g., {\tt psBe 0.3 -0.1 3.1} --- The tag must begin with {\tt ps} (case insensitive).  It is used to put a pseudopotential on selected atoms ({\em I think ???????}).


\chapter{Ground state molecular DFT and HF}

\begin{verbatim}
  dft
      directives/keywords
  end
\end{verbatim}

{\tt nopen value} --- E.g., {\tt nopen 3} --- The number of unpaired spin orbitals, $n_\alpha - n_\beta$ (default {\tt 0})

{\tt unrestricted} --- Selects a spin-unrestricted calculation (default is spin restricted)

{\tt xc value} --- E.g., {\tt xc HF} --- Selects the exchange correlation potential (default is {\tt LDA}). See XC section for more details.

{\tt aobasis value} --- E.g., {\tt aobasis sto-3g} --- Sets the atomic orbital basis used for the initial guess.  Options are {\tt sto-3g} (down to Iodine)  or {\tt 6-31g} (down to Zinc, default).  

{\tt charge value} --- E.g., {\tt charge -1.0} --- Total charge (default {\tt 0}) on the molecule. Atomic units.

{\tt nvalpha value} --- E.g., {\tt nvalpha 2} --- The number of alpha spin virtual orbitals to solve for (default {\tt 0}) --- is this working now?

{\tt nvbeta value} --- E.g., {\tt nvbeta 2} --- The number of beta spin virtual orbitals to solve for (default {\tt 0}) --- is this working now?

{\tt no\_orient} --- Do not reorient/translate the molecule to orientation/center.

{\tt core\_type value} --- E.g., {\tt What is available?} Selects the pseudopotential to be used on all atoms (can also do mixed all-electron/pseudopotential calculation).  Not heavily tested and unoptimized. (default is all electron).

{\tt psp\_calc} --- Perform pseusopotential calculation on all atoms.  Not heavily tested and unoptimized. (default is all-electron)

{\tt L value} --- E.g., {\tt L 50} --- Sets the computational box size to $[-L,L]^3$ atomic units (mostly for testing).  Default is to find cube that contains all nuclei and to add 50 atomic units.

\section{XC --- DFT exchange correlation}

Without LIBXC, the code just provides either Hartree-Fock ({\tt xc HF}) or local (spin) density approximation ({\tt xc LDA}, the default).

With LIBXC, in addition to HF and LDA (default) there are wide varity of GGA and hybrid functionals available --- the ones that have been tested (to some extent) have been provided with simple input formats
\begin{itemize}
\item Becke-Perdew (91?) --- \verb+xc bp+
\item Becke-Perdew 86 --- \verb+xc bp86+
\item PBE --- \verb+xc pbe+
\item PBE-0 --- \verb+xc pbe0+
\item B3LYP --- \verb+xc b3lyp+
\end{itemize}

There is also a more general input format in which you can provide a list of functional names (using LIBXC's naming scheme) and the coefficient by which to scale it.  For instance, to get PBE-0 you could specify
\verb+xc GGA_X_PBE .75 GGA_C_PBE 1. HF_X .25+.

We have not yet implemented the near linear-scaling algorithm for HF
exchange, which as a consequence is fairly slow and may require a lot
of memory (there is low-memory algorithm in the code but I don't think
it is yet available as an input option).

\section{Restarting}

At completion of an HF or DFT calculation, the molecular orbitals are
saved in the files \verb+restartdata.*+ (with one file per I/O server
process).  The projection of the orbtials onto the {\tt sto-3G} AO basis
set is saved into the file {\tt restartaodata}.

{\tt restart} --- Restart from numerical orbitals from a previous calculation (default is no)

{\tt restartao} --- Restart from projection of orbitals onto AO basis
set from a previous calculation (default is no unless doing geometry
optimization).  If a restart file is not found, or the file contains
incompatible data then the default atomic guess is used.

{\tt save value} --- E.g., {\tt save false} --- Boolean flag to save (or not) orbitals at completion (default is true).

\section{Controlling convergence and accuracy}

The default convergence test is on both the 2-norm of change in density per atom (separately for each spin) between iterations and the residual error in each wave function.

\begin{verbatim}
  converged = (da < dconv * molecule.natom()) && 
              (db < dconv * molecule.natom()) &&
              (conv_only_dens || (max_residual < 5.0 * dconv))
\end{verbatim}

{\tt dconv value} --- E.g., {\tt dconv 1e-5} --- SCF convergence criterion (default 1e-4 atomic units).  Suggest decreasing this to {\tt 1e-5} for geometry optimization or property calculations.

{\tt canon} --- Solves for canonical orbitals or eigenfunctions (default is localized orbitals except for atoms and diatomics).

{\tt pm} --- Selects use of the Pipek-Mezy localized orbitals (default).

{\tt boys} --- Selects use of the Boys localized orbitals.

{\tt maxrotn value} --- E.g., {\tt maxrotn 0.1} --- Used to restrict maximum rotation of orbitals (default {\tt 0.25})

{\tt maxiter value} --- E.g., {\tt maxiter 20} --- The maximum number of iterations (default is {\tt 20})

{\tt maxsub value} --- E.g., {\tt maxsub 5} --- The size of the iterative subspace (default is {\tt 5}).  Sometimes it helps to make this larger.

{\tt protocol valuelist} --- E.g., {\tt protocol 1e-4 1e-6 1e-8} --- Sets the solution protocol.  The default is `{\tt 1e-4 1e-6} which means solve first using a truncation threshold of {\tt 1e-4} (using $k=6$) and with a threshold of {\tt 1e-6} (using $k=8$).

{\tt orbitalshift} --- E.g., {\tt orbitalshift 0.1} --- Shifts the occupied orbitals down in energy by the given amount (default {\tt 0}).  Is this working?

{\tt k value} --- E.g., {\tt k 8} --- Sets the wavelet order to a fixed value (mostly only used for testing)

{\tt convonlydens} --- Just test on the change in the density for convergence.


\section{Geometry optimization}

By default geometry optimization is performed using the BFGS Hessian update algorithm.  The convergence test is on all of the 2-norm of the gradient, the change in the energy between iterations, and the maximum change in Cartesian coordinates (all in atomic units).  The 

For geometry optimization it is recommended to select {\tt dconv 1e-5} to obtain more accurate gradients.

{\tt gopt} --- Requests optimization of the geometry

{\tt gtol value} --- E.g., {\tt gtol 1e-4} --- Sets the convergence threshold for the 2-norm of the gradient (default {\tt 1e-3}).

{\tt gtest value} --- E.g., {\tt gtest 1e-4} --- Sets the convergence threshold for the maximum change in Cartesian coordinates (default {\tt 1e-3} atomic units).

{\tt gval value} --- E.g., {\tt gval 1e-6} --- Sets the available precision in the energy (default is {\tt 1e-5} atomic units).

{\tt gprec value} --- E.g., {\tt gtest 1e-6} --- Sets the available precision in the gradient (default is {\tt 1e-5} atomic units).

{\tt gmaxiter value} --- E.g., {\tt gmaxiter 100} --- Sets the maximum number of geometry optimization iterations (default is 20).

{\tt algopt value} --- E.g., {\tt algopt SR1} --- Selects the quasi-Newton update method (default is ).  Options are {\tt BFGS} (default) or {\tt SR1} (not heavily tested).  Case sensitive.

\section{Properties}

{\tt derivatives} --- Compute the derivates (default is false).

{\tt dipole} --- Compute the molecular dipole moment (default is false --- why?).

{\tt response} --- TBD

{\tt response\_freq} ---  TBD

{\tt response\_axis} --- TBD

{\tt rconv} --- TBD

{\tt efield} --- TBD

{\tt efield\_axis x y z} --- TBD

{\tt print\_dipole\_matels} --- TBD

\section{Plotting}

Plots are generated to OpenDX files.  In the {\tt moldft} source directory are two useful files

\begin{itemize}
\item {\tt vizit.net} --- An OpenDX visual program that displays a
  molecule (from file {\tt molecule.dx}) along with positive+negative
  isosurfaces (with adjustable value)for a scalar field read from a
  file.

\item {\tt moldx.py} --- A Python program you can run with your {\em
  output} file as standard input to produce a {\tt molecule.dx} file.
  It is important to use your output file since {\tt moldft} will (by
  default) translate and rotate the molecular coordinates.
  
\end{itemize}

{\tt plotmos lo hi} --- E.g., {\tt plotmos 10 12} --- Plots the molecular orbitals in the given inclusive range (default is none).  Orbitals are numbered from zero.  Seems like this needs extending to accomodate unrestricted calculations.

{\tt plotdens} --- Plots the total electronic charge density and, if spin unrestricted, the spin density (default is off).

{\tt plotcoul} --- Plots the total (electronic + nuclear) electrostatic potential (default is off).

{\tt plotnpt value } --- E.g., {\tt plotnpt 501} --- Sets the number of plots used per dimension in the cube of points (default 101).

{\tt plotcell xlo xhi ylo yhi zlo zhi} --- E.g., {\tt plotcell -10 10 -15 15 -10 5} --- Sets the cell (in atomic units) used for plotting (default is the entire simulation cell).

\section{Parallel execution}

{\tt loadbal vnucfac parts} --- E.g., {\tt loadbal 12 2} --- Adjusts data/loadbalance when running in parallel with MPI.  {\tt vnucfac} (default 12) is extra weight associated with nuclear potential and {\tt parts} (default 2) is the number of partitions (or subtrees) per node.  SCF

{\tt nio value} --- E.g., {\tt nio 10} --- The number of MPI processes to use as I/O servers (default is 1)


\end{document}
