# MPI Multi-Threaded (MPI-MT)

These are notes describing the MPI multi-threaded runtime. These notes are intended to help developers navigate the contents of these files and to locate specific functionality.

The MPI-MT is intended to be high-performing but experience tells us that it is not.  It relies on the MPI-2 standard to allow `MPI_Init_thread` with a threading level of `MPI_THREAD_MULTIPLE`.  Every MPI rank creates a thread that simply calls the data server progress engine indefinitely.  The progress thread blocks in an `MPI_Recv` with `MPI_ANY_SOURCE` and handles incoming requests to the data server.  User-level requests, e.g., `comex_put`, make progress at the initiator by internally waiting for an available non-blocking handle.  By default, all functions call an internal non-blocking interface.  The blocking calls, e.g., `comex_get`, internally call a non-blocking get followed by a blocking wait.  The user-level non-blocking handles are also flushed any time a `comex_barrier` is called, since it enforces a communication fence followed by a barrier.

There are a finite number of user-level non-blocking handles.  This is set using the environment variable `COMEX_MAX_NB_OUTSTANDING`.  This controls the size of an allocated array of our non-blocking handle data structure `nb_t`.  The `nb_t` structure contains linked lists of `MPI_Request` objects associated with the given user-level handle.  It is slightly more complicated than that since get requests might be using the packing optimization where the request is first compressed into a contiguous buffer.  The stride information is kept with the `nb_t` message so that the received buffer can be unpacked.  All memory is freed when operations complete.

Incoming requests are all based on the active message concept.  A 'header' message is sent first to the progress engine indicating the type of request, e.g., `OP_PUT`, `OP_ACC_INT`.  The header contains enough information to complete the request such as source and destination pointers, source and destination ranks, etc.  After a header message is sent, any data payload is sent as a separate message.  The intent was to let MPI directly use the buffer pointers in case the buffers were allocated using any special, network-specific allocator.  Otherwise, a data payload could have been mem-copied to the end of the header message (this is done in the MPI-PR implementation as an optimization).
