
==========
1.0 BUILD:
==========

The default build includes a non-debug version of libdat and libdapl-cma uDAPL provider. It will also builds test suites dtest and dapltest and provides manpages for each. This version requires libibverbs and librdmacm installation, IPoIB installation, and IPoIB configuration with an IP address. 

Building :
----------
./autogen.sh 
./configure 
make

Building debug version:
----------------------
./autogen.sh
./configure --enable-debug
make

Build example with OFED prefix (x86_64)
---------------------------------------------
./autogen.sh
./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
make

Installing:
----------
make install

NOTE: to link these libraries you must either use libtool and 
specify the full pathname of the library, or use the `-LLIBDIR' 
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,--rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

===================
2.0 CONFIGURATION:
===================

/etc/dat.conf 

# DAT v2.0
#
# Note: Both API and Provider versions are based on DAT specification, v2.0
#  
# Each entry should have the following fields:
#
# <ia_name> <api_version> <threadsafety> <default> <lib_path> \
#           <provider_version> <ia_params> <platform_params>
#
# For uDAPL cma provder, <ia_params> is one of the following:
#       network address, network hostname, or netdev name and 0 for port
#
# For uDAPL scm provider, <ia_params> is device name and port
# For uDAPL ucm provider, <ia_params> is device name and port
# For uDAPL iWARP provider, <ia_params> is netdev device name and 0 
# For uDAPL iWARP provider, <ia_params> is netdev device name and 0 
# For uDAPL RoCE provider, <ia_params> is device name and 0 
# 
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib0 0" ""
ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib1 0" ""
ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 1" ""
ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 2" ""
ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 1" ""
ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 2" ""
ofa-v2-ehca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ehca0 1" ""
ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-mthca0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mthca0 1" ""
ofa-v2-mthca0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mthca0 2" ""
ofa-v2-cma-roe-eth2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
ofa-v2-cma-roe-eth3 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth3 0" ""
ofa-v2-scm-roe-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-scm-roe-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-mcm-1 u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mcm-2 u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-scif0 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "scif0 1" ""
ofa-v2-scif0-u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "scif0 1" ""
ofa-v2-mic0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "mic0:ib 1" ""
ofa-v2-mlx4_0-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-mlx4_1-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_1 1" ""
ofa-v2-mlx4_1-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_1 2" ""
ofa-v2-mlx4_1-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_1 1" ""
ofa-v2-mlx4_1-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_1 2" ""
ofa-v2-mlx4_0-1m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-mlx4_1-1m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_1 1" ""
ofa-v2-mlx4_1-2m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_1 2" ""
ofa-v2-mlx5_0-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx5_0 1" ""
ofa-v2-mlx5_0-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx5_0 2" ""
ofa-v2-mlx5_1-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx5_1 1" ""
ofa-v2-mlx5_1-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx5_1 2" ""
ofa-v2-mlx5_0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_0 1" ""
ofa-v2-mlx5_0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_0 2" ""
ofa-v2-mlx5_1-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_1 1" ""
ofa-v2-mlx5_1-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_1 2" ""
ofa-v2-mlx5_0-1m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx5_0 1" ""
ofa-v2-mlx5_0-2m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx5_0 2" ""
ofa-v2-mlx5_1-1m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx5_1 1" ""
ofa-v2-mlx5_1-2m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx5_1 2" ""
ofa-v2-qib0-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "qib0 1" ""
ofa-v2-qib0-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "qib0 2" ""
ofa-v2-qib1-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "qib1 1" ""
ofa-v2-qib1-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "qib1 2" ""

=============================
3.0 Environment Variables
=============================

 - IB UD options using UCM provider, large scale settings (Xeon)
 
export DAPL_UCM_REP_TIME=2000   /*  REQUEST timer, waiting on REPLY, msecs, default = 800 */
export DAPL_UCM_RTU_TIME=1000   /* REPLY timer, waiting for RTU in msecs, default=400 */
export DAPL_UCM_RETRY=7       	/* REQUEST & REPLY retries, default = 7 */
export DAPL_UCM_QP_SIZE=1000	/* CM req/reply work queue size, default = 500 entries */
export DAPL_UCM_CQ_SIZE=1000	/* CM req/reply completion queue size, default = 500 entries */
export DAPL_UCM_TX_BURST=100	/* CM signal rate on send messages */

 - IB RC options using UCM provider, large scale settings (Xeon)
 
export DAPL_MAX_INLINE=64	/*  IB RC inline optimization, best small msg latency, def=64 */
export DAPL_ACK_RETRY=7         /*  IB RC Ack retry count, default 7 */
export DAPL_ACK_TIMER=20       	/* IB RC Ack retry timer, default 20 */
					IB formula:: 5 bits, 4.096us*2^ack_timer. 16== 268ms, 20==4.2s
- IB RC options using SCM provider

export DAPL_SCM_NETDEV=mic0	/* default is first non-loopback netdev,  use mic0 with KNCs */

Other  IB settings:
export DAPL_IB_MTU=2048		/* IB MTU size, default = 2048 */
export DAPL_RNR_TIMER=12	/* 5 bits, 12 =.64ms, 28 =163ms, 31 =491ms */
export DAPL_RNR_RETRY=7		/* 3 bits, 7 == infinite */
export DAPL_IB_PKEY= 0		/* override IB partition key, default is pkey index 0 */
export DAPL_IB_SL=0		/* override IB Sevice level, default = 0 */

- Other options:
export DAPL_WR_MAX=500 		/* used to reduce max qp depth on all IB providers, default = dev attributes */

Debug logging and Counter settings ( --enable-counters)

export DAPL_DBG_SYS_MEM=10	/* threshold for low sys memory warning, def = 10 percent */
export DAPL_DBG_TYPE=0x0000003 	/* set log, monitor, and error checking, default = warnings and errors */

DAPL_DBG_TYPE bit settings as follow:

DAPL_DBG_TYPE_ERR          = 0x0001,
DAPL_DBG_TYPE_WARN         = 0x0002,
DAPL_DBG_TYPE_EVD          = 0x0004,
DAPL_DBG_TYPE_CM           = 0x0008,
DAPL_DBG_TYPE_EP           = 0x0010,
DAPL_DBG_TYPE_UTIL         = 0x0020,
DAPL_DBG_TYPE_CALLBACK     = 0x0040,
DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080,
DAPL_DBG_TYPE_API          = 0x0100,
DAPL_DBG_TYPE_RTN          = 0x0200,
DAPL_DBG_TYPE_EXCEPTION   = 0x0400,
DAPL_DBG_TYPE_SRQ         = 0x0800,
DAPL_DBG_TYPE_CNTR        = 0x1000,
DAPL_DBG_TYPE_CM_LIST     = 0x2000,
DAPL_DBG_TYPE_THREAD      = 0x4000,
DAPL_DBG_TYPE_CM_EST      = 0x8000,
DAPL_DBG_TYPE_CM_WARN    = 0x10000,
DAPL_DBG_TYPE_EXTENSION  = 0x20000,
DAPL_DBG_TYPE_CM_STATS   = 0x40000,
DAPL_DBG_TYPE_CM_ERRS    = 0x80000,    /* print any cm errors on device close */
DAPL_DBG_TYPE_LINK_ERRS  = 0x100000,   /* print any link errors on device close */
DAPL_DBG_TYPE_LINK_WARN  = 0x200000,   /* print any link warning on device close */
DAPL_DBG_TYPE_DIAG_ERRS  = 0x400000,   /* print any diag_counter errors on dev close */
DAPL_DBG_TYPE_SYS_WARN   = 0x800000,   /* print low mem warning during alloc, reg_mem */
DAPL_DBG_TYPE_VER        = 0x1000000,  /* print dapl ver and build date during dev open */


=============================
4.0 Bugs/Known issues
=============================

mlx5 support included, new dat.conf entries. beta level.

=============================
5.0 SAMPLE uDAPL APPLICATION:
=============================

There are 2 sample programs, with manpages, provided with this package.

(dapl/test/dtest/)

NAME
       dtest - simple uDAPL send/receive and RDMA test

SYNOPSIS
       dtest [-P provider] [-b buf size] [-B burst count][-v] [-c] [-p] [-d] [-s]

       dtest [-P provider] [-b buf size] [-B burst count][-v] [-c] [-p] [-d] [-h HOSTNAME]

DESCRIPTION
       dtest  is a simple test used to exercise and verify the uDAPL interfaces.  At least two instantia-
       tions of the test must be run. One acts as the server and the other the client. The server side of
       the  test,  once invoked listens for connection requests, until timing out or killed. Upon receipt
       of a cd connection request, the connection is established, the server and  client  sides  exchange
       information necessary to perform RDMA writes and reads.

OPTIONS
       -P=PROVIDER
              use PROVIDER to specify uDAPL interface using /etc/dat.conf (default OpenIB-cma)

       -b=BUFFER_SIZE
              use buffer size BUFFER_SIZE for RDMA(default 64)

       -B=BURST_COUNT
              use busrt count BURST_COUNT for interations (default 10)

       -v, verbose output(default off)

       -c, use consumer notification events (default off)

       -p, use polling (default wait for event)

       -d, delay in seconds before close (default off)

       -s, run as server (default - run as server)

       -h=HOSTNAME
              use HOSTNAME to specify server hostname or IP address (default - none)

EXAMPLES
       dtest -P OpenIB-cma -v -s
            Starts a server process with debug verbosity using provider OpenIB-cma.

       dtest -P OpenIB-cma -h server1-ib0

            Starts a client process, using OpenIB-cma provider to connect to hostname server1-ib0.

SEE ALSO
       dapltest(1)

AUTHORS
       Arlin Davis
              <ardavis@ichips.intel.com>

BUGS

/dapl/test/dapltest/

NAME
        dapltest - test for the Direct Access Programming Library (DAPL)

DESCRIPTION
       Dapltest  is  a  set  of tests developed to exercise, characterize, and verify the DAPL interfaces
       during development and porting.  At least two instantiations of the test must be run. One acts  as
       the  server, fielding requests and spawning server-side test threads as needed. Other client invo-
       cations connect to the server and issue test requests. The server side of the test, once  invoked,
       listens  continuously for client connection requests, until quit or killed. Upon receipt of a con-
       nection request, the connection is established, the server and client sides swap  version  numbers
       to  verify that they are able to communicate, and the client sends the test request to the server.
       If the version numbers match, and the test request is well-formed, the server spawns  the  threads
       needed to run the test before awaiting further connections.

USAGE
       dapltest [ -f script_file_name ] [ -T S|Q|T|P|L ] [ -D device_name ] [ -d ] [ -R HT|LL|EC|PM|BE ]

       With  no  arguments,  dapltest runs as a server using default values, and loops accepting requests
       from clients.

       The -f option allows all arguments to be placed in a file, to ease test automation.

       The following arguments are common to all tests:

       [ -T S|Q|T|P|L ]
              Test function to be performed:

              S      - server loop

              Q      - quit, client requests that server wait for any outstanding tests to complete, then
                     clean up and exit

              T      - transaction test, transfers data between client and server

              P      - performance test, times DTO operations

              L      -  limit  test,  exhausts  various  resources, runs in client w/o server interaction
                     Default: S

      [ -D device_name ]
              Specifies the interface adapter name as documented in the /etc/dat.conf  static  configura-
              tion file. This name corresponds to the provider library to open.  Default: none

       [ -d ] Enables  extra  debug  verbosity,  primarily tracing of the various DAPL operations as they
              progress.  Repeating this parameter increases debug spew.  Errors encountered result in the
              test  spewing some explanatory text and stopping; this flag provides more detail about what
              lead up to the error.  Default: zero

       [ -R BE ]
              Indicate the quality of service (QoS) desired.  Choices are:

              HT     - high throughput

              LL     - low latency

              EC     - economy (neither HT nor LL)

              PM     - premium

              BE     - best effort Default: BE

       Usage - Quit test client

           dapltest [Common_Args] [ -s server_name ]

           Quit testing (-T Q) connects to the server to ask it to clean up and
           exit (after it waits for any outstanding test runs to complete).
           In addition to being more polite than simply killing the server,
           this test exercises the DAPL object teardown code paths.
           There is only one argument other than those supported by all tests:

           -s server_name      Specifies the name of the server interface.
                               No default.

       Usage - Transaction test client

           dapltest [Common_Args] [ -s server_name ]
                    [ -t threads ] [ -w endpoints ] [ -i iterations ] [ -Q ]
                    [ -V ] [ -P ] OPclient OPserver [ op3,

           Transaction testing (-T T) transfers a variable amount of data between
:
          client and server.  The data transfer can be described as a sequence of
           individual operations; that entire sequence is transferred ’iterations’
           times by each thread over all of its endpoint(s).

           The following parameters determine the behavior of the transaction test:

           -s server_name      Specifies the name or IP address of the server interface.
                               No default.

           [ -t threads ]      Specify the number of threads to be used.
                               Default: 1

           [ -w endpoints ]    Specify the number of connected endpoints per thread.
                               Default: 1

           [ -i iterations ]   Specify the number of times the entire sequence
                               of data transfers will be made over each endpoint.
                               Default: 1000

           [ -Q ]              Funnel completion events into a CNO.
                               Default: use EVDs

           [ -V ]              Validate the data being transferred.
                               Default: ignore the data

           [ -P ]              Turn on DTO completion polling
                               Default: off

           OP1 OP2 [ OP3, ... ]
                               A single transaction (OPx) consists of:

                               server|client   Indicates who initiates the
                                               data transfer.

                               SR|RR|RW        Indicates the type of transfer:
                                               SR  send/recv
                                               RR  RDMA read
                                               RW  RDMA write
                               Defaults: none

                               [ seg_size [ num_segs ] ]
:

                                              Indicates the amount and format
                                               of the data to be transferred.
                                               Default:  4096  1
                                                         (i.e., 1 4KB buffer)

                               [ -f ]          For SR transfers only, indicates
                                               that a client’s send transfer
                                               completion should be reaped when
                                               the next recv completion is reaped.
                                               Sends and receives must be paired
                                               (one client, one server, and in that
                                               order) for this option to be used.

           Restrictions:

           Due to the flow control algorithm used by the transaction test, there
           must be at least one SR OP for both the client and the server.

           Requesting data validation (-V) causes the test to automatically append
           three OPs to those specified. These additional operations provide
           synchronization points during each iteration, at which all user-specified
           transaction buffers are checked. These three appended operations satisfy
           the "one SR in each direction" requirement.

           The transaction OP list is printed out if -d is supplied.

       Usage - Performance test client

           dapltest [Common_Args] -s server_name [ -m p|b ]
                    [ -i iterations ] [ -p pipeline ] OP

           Performance testing (-T P) times the transfer of an operation.
           The operation is posted ’iterations’ times.

           The following parameters determine the behavior of the transaction test:

           -s server_name      Specifies the name or IP address of the server interface.
                               No default.

           -m b|p              Used to choose either blocking (b) or polling (p)
                               Default: blocking (b)
          [ -i iterations ]   Specify the number of times the entire sequence
                               of data transfers will be made over each endpoint.
                               Default: 1000

           [ -p pipeline ]     Specify the pipline length, valid arguments are in
                               the range [0,MAX_SEND_DTOS]. If a value greater than
                               MAX_SEND_DTOS is requested the value will be
                               adjusted down to MAX_SEND_DTOS.
                               Default: MAX_SEND_DTOS

           OP                  Specifies the operation as follow:

                               RR|RW           Indicates the type of transfer:
                                               RR  RDMA read
                                               RW  RDMA write
                                               Defaults: none

                               [ seg_size [ num_segs ] ]
                                               Indicates the amount and format
                                               of the data to be transferred.
                                               Default:  4096  1
                                                         (i.e., 1 4KB buffer)

       Usage - Limit test client

           Limit testing (-T L) neither requires nor connects to any server
           instance.  The client runs one or more tests which attempt to
           exhaust various resources to determine DAPL limits and exercise
           DAPL error paths.  If no arguments are given, all tests are run.

           Limit testing creates the sequence of DAT objects needed to
           move data back and forth, attempting to find the limits supported
           for the DAPL object requested.  For example, if the LMR creation
           limit is being examined, the test will create a set of
           {IA, PZ, CNO, EVD, EP} before trying to run dat_lmr_create() to
           failure using that set of DAPL objects.  The ’width’ parameter
           can be used to control how many of these parallel DAPL object
           sets are created before beating upon the requested constructor.
           Use of -m limits the number of dat_*_create() calls that will
           be attempted, which can be helpful if the DAPL in use supports
           essentailly unlimited numbers of some objects.
           The limit test arguments are:

           [ -m maximum ]      Specify the maximum number of dapl_*_create()
                               attempts.
                               Default: run to object creation failure

           [ -w width ]        Specify the number of DAPL object sets to
                               create while initializing.
                               Default: 1

           [ limit_ia ]        Attempt to exhaust dat_ia_open()

           [ limit_pz ]        Attempt to exhaust dat_pz_create()

           [ limit_cno ]       Attempt to exhaust dat_cno_create()

           [ limit_evd ]       Attempt to exhaust dat_evd_create()

           [ limit_ep ]        Attempt to exhaust dat_ep_create()

           [ limit_rsp ]       Attempt to exhaust dat_rsp_create()

           [ limit_psp ]       Attempt to exhaust dat_psp_create()

           [ limit_lmr ]       Attempt to exhaust dat_lmr_create(4KB)

           [ limit_rpost ]     Attempt to exhaust dat_ep_post_recv(4KB)

           [ limit_size_lmr ]  Probe maximum size dat_lmr_create()

                               Default: run all tests

EXAMPLES
       dapltest -T S -d -D OpenIB-cma

                               Starts a server process with debug verbosity.

       dapltest -T T -d -s host1-ib0 -D OpenIB-cma -i 100 client SR 4096 2 server SR 4096 2

                               Runs a transaction test, with both sides
                               sending one buffer with two 4KB segments,
                              one hundred times.

       dapltest -T P -d -s host1-ib0 -D OpenIB-cma -i 100 SR 4096 2

                               Runs a performance test, with the client
                               sending one buffer with two 4KB segments,
                               one hundred times.

       dapltest -T Q -s host1-ib0 -D OpenIB-cma

                               Asks the server to clean up and exit.

       dapltest -T L -D OpenIB-cma -d -w 16 -m 1000

                               Runs all of the limit tests, setting up
                               16 complete sets of DAPL objects, and
                               creating at most a thousand instances
                               when trying to exhaust resources.

       dapltest -T T -V -d -t 2 -w 4 -i 55555 -s linux3 -D OpenIB-cma client RW 4096 1 server RW  2048  4
       client SR 1024 4 server SR 4096 2 client SR 1024 3 -f server SR 2048 1 -f

                               Runs a more complicated transaction test,
                               with two thread using four EPs each,
                               sending a more complicated buffer pattern
                               for a larger number of iterations,
                               validating the data received.

       BUGS   (and  To Do List)

           Use of CNOs (-Q) is not yet supported.

           Further limit tests could be added.




