                    ARMCI on MPI-RMA Implementation Notes
                       James Dinan <dinan@mcs.anl.gov>

===============================================================================
Introduction
===============================================================================

This project provides a full, high performance, portable implementation of the
ARMCI runtime system using MPI's remote memory access (RMA) functionality.

===============================================================================
Installing ARMCI-MPI
===============================================================================

ARMCI-MPI uses autoconf and must be configured before compiling:

 $ ./configure

Many configure options are provided, run "configure --help" for details.  After
configuring the source tree, the code can be built and installed by running:

 $ make && make install

ARMCI-MPI can be used with GA 5.0 and later by substituting this directory for
the "armci" directory in the GA distribution.  The quality of MPI-RMA
implementations varies.  As of August, 2011 the following MPI implementations are
known to work correctly with ARMCI-MPI:

 + MVAPICH2 1.6
 + MPICH
 + Cray MPI on Cray XE6
 + IBM MPI on BG/P (set ARMCI_STRIDED_METHOD=IOV and ARMCI_IOV_METHOD=BATCHED)
 + OpenMPI 1.5.4 (set ARMCI_STRIDED_METHOD=IOV and ARMCI_IOV_METHOD=BATCHED)

The following MPI implementations are known to fail with ARMCI-MPI:

 - MVAPICH2 prior to 1.6

===============================================================================
The ARMCI-MPI Test Suite
===============================================================================

ARMCI-MPI includes a set of testing and benchmark programs located under tests/
and benchmarks/.  These programs can be compiled and run via:

$ make check MPIEXEC="mpiexec -n 4"

The MPIEXEC variable is optional and is used to override the default MPI launch
command.  If you want only to build the test suite, the following target can be
used:

$ make checkprogs

===============================================================================
ARMCI-MPI Errata
===============================================================================

Direct access to local buffers:

 * Because of MPI's semantics, you are not allowed to access shared memory
   directly, it must be through put/get.  Alternatively you can use the 
   new ARMCI_Access_begin/end() functions.
   
Progress semantics:

 * On some MPI implementations and networks you may need to enable implicit
   progress.  In many cases this is done through an environment variable.  For
   MPICH: set MPICH_ASYNC_PROGRESS; for MVAPICH2 recompile with
   --enable-async-progress and set MPICH_ASYNC_PROGRESS; set DCMF_INTERRUPTS=1
   for MPICH-BG; etc.

===============================================================================
Environment Variables:
===============================================================================

Boolean environment variables are enabled when set to a value beginning with
't', 'T', 'y', 'Y', or '1'; any other value is interpreted as false.

 -------------------
: Debugging Options :
 -------------------

ARMCI_VERBOSE (boolean)

  Enable extra status output from ARMCI-MPI.

ARMCI_DEBUG_ALLOC (boolean)

  Turn on extra shared allocation debugging.

ARMCI_FLUSH_BARRIERS (boolean)

  Enable/disable extra communication flushing in ARMCI_Barrier.  Extra flushes
  are present to help make unsafe DLA safer.

 ---------------------
: Performance Options :
 ---------------------

ARMCI_CACHE_RANK_TRANSLATION (boolean)

  Create a table to more quickly translate between absolute and group ranks.

 --------------------------
: Noncollective Groups     :
 --------------------------

ARMCI_NONCOLLECTIVE_GROUPS (boolean)

  Enable noncollective ARMCI group formation; group creation is collective on
  the output group rather than the parent group.

 --------------------------
: Shared Buffer Protection :
 --------------------------

ARMCI_SHR_BUF_METHOD = { COPY (default), NOGUARD }

  ARMCI policy for managing shared origin buffers in communication operations:
  lock the buffer (unsafe, but fast), copy the buffer (safe), or don't guard
  the buffer - assume that the system is cache coherent and MPI supports
  unlocked load/store.

 --------------------
: I/O Vector Options :
 --------------------

ARMCI_IOV_METHOD = { AUTO (default), CONSRV, BATCHED, DIRECT }

  Select the IO vector communication strategy: automatic; a "conservative"
  implementation that does lock/unlock around each operation; an implementation
  that issues batches of operations within a single lock/unlock epoch; and a
  direct implementation that generates datatypes for the origin and target and
  issues a single operation using them.

ARMCI_IOV_CHECKS (boolean)

  Enable (expensive) IOV safety/debugging checks (not recommended for
  performance runs).

ARMCI_IOV_BATCHED_LIMIT = { 0 (default), 1, ... }

  Set the maximum number of one-sided operations per epoch for the BATCHED IOV
  method.  Zero (default) is unlimited.
  
 -----------------
: Strided Options :
 -----------------

ARMCI_STRIDED_METHOD = { DIRECT (default), IOV }

  Select the method for processing strided operations.
