# -*- text -*-
#
# Copyright (c) 2020      Intel, Inc.  All rights reserved.
# Copyright (c) 2020      IBM Corporation.  All rights reserved.
# Copyright (c) 2021-2026 Nanook Consulting  All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
#
[rtos]
The "--rtos" command line directive must be accompanied by
a comma-delimited list of case-insensitive options that control the
runtime behavior of the job. The full directive need not be provided —
only enough characters are required to uniquely identify the
directive.

Runtime options are typically "true" or "false", though this is not a
requirement on developers. Since the value of each option may need to
be set (e.g., to override a default set by MCA parameter), the syntax
of the command line directive includes the use of an "=" character to
allow inclusion of a value for the option. For example, one can set
the "ERROR-NONZERO-STATUS" option to "true" by specifying it as
"ERROR-NONZERO-STATUS=1". Note that boolean options can be set to
"true" using a non-zero integer or a case-insensitive string of the
word "true".  For the latter representation, the user need only
provide at least the "T" character. The same policy applies to setting
a boolean option to "false".

Note that a boolean option will default to "true" if provided without
a value. Thus, "--runtime-options error-nonzero" is sufficient to set
the "ERROR-NONZERO-STATUS" option to "true".

Supported values include:

* "ERROR-NONZERO-STATUS[=(bool)]": if set to false, this directs the
  runtime to treat a process that exits with non-zero status as a
  normal termination.  If set to true, the runtime will consider such
  an occurrence as an error termination and take appropriate action —
  i.e., the job will be terminated unless a runtime option directs
  otherwise. This option defaults to a true value if the option is
  given without a value.

* "DONOTLAUNCH": directs the runtime to map but not launch the
  specified job. This is provided to help explore possible process
  placement patterns before actually starting execution. No value need
  be passed as this is not an option that can be set by default in
  PRRTE.

* "SHOW-PROGRESS[=(bool)]": requests that the runtime provide progress
  reports on its startup procedure — i.e., the launch of its daemons
  in support of a job. This is typically used to debug DVM startup on
  large systems.  This option defaults to a true value if the option
  is given without a value.

* "NOTIFYERRORS[=(bool)]": if set to true, requests that the runtime
  provide a PMIx event whenever a job encounters an error — e.g., a
  process fails.  The event is to be delivered to each remaining
  process in the job. This option defaults to a true value if the
  option is given without a value.  See "--help notifications" for
  more detail as to the PMIx event codes available for capturing
  failure events.

* "RECOVERABLE[=(bool)]": if set to true, this indicates that the
  application wishes to consider the job as recoverable — i.e., the
  application is assuming responsibility for recovering from any
  process failure. This could include application-driven spawn of a
  substitute process or internal compensation for the missing process.
  This option defaults to a true value if the option is given without
  a value.

* "AUTORESTART[=(bool)]": if set to true, this requests that the
  runtime automatically restart failed processes up to "max restarts"
  number of times. This option defaults to a true value if the option
  is given without a value.

* "CONTINUOUS[=(bool)]": if set to true, this informs the runtime that
  the processes in this job are to run until explicitly terminated.
  Processes that fail are to be automatically restarted up to "max
  restarts" number of times. Notification of process failure is to be
  delivered to all processes in the application. This is the
  equivalent of specifying "RECOVERABLE", "NOTIFYERRORS", and
  "AUTORESTART" options except that the runtime, not the application,
  assumes responsibility for process recovery. This option defaults to
  a true value if the option is given without a value.

* "MAX-RESTARTS=<int>": indicates the maximum number of times a given
  process is to be restarted. This can be set at the application or
  job level (which will then apply to all applications in that job).

* "EXEC-AGENT=<path>" indicates the executable that shall be used to
  start an application process. The resulting command for starting an
  application process will be "<path> app <app-argv>". The path may
  contain its own command line arguments.

* "DEFAULT-EXEC-AGENT": directs the runtime to use the system default
  exec agent to start an application process. No value need be passed
  as this is not an option that can be set by default in PRRTE.

* "OUTPUT-PROCTABLE[(=channel)]": directs the runtime to report the
  convential debugger process table (includes PID and host location of
  each process in the application). Output is directed to stdout if
  the channel is "-", stderr if "+", or into the specified file
  otherwise. If no channel is specified, output will be directed to
  stdout.

* "STOP-ON-EXEC": directs the runtime to stop the application
  process(es) immediately upon exec'ing them. The directive will apply
  to all processes in the job.

* "STOP-IN-INIT": indicates that the runtime should direct the
  application process(es) to stop in "PMIx_Init()". The directive will
  apply to all processes in the job.

* "STOP-IN-APP": indicates that the runtime should direct application
  processes to stop at some application-defined place and notify they
  are ready-to-debug. The directive will apply to all processes in the
  job.

* "TIMEOUT=<string>": directs the runtime to terminate the job after
  it has executed for the specified time. Time is specified in colon-
  delimited format — e.g., "01:20:13:05" to indicate 1 day, 20 hours,
  13 minutes and 5 seconds. Time specified without colons will be
  assumed to have been given in seconds.

* "SPAWN-TIMEOUT=<string>": directs the runtime to terminate the job
  if job launch is not completed within the specified time. Time is
  specified in colon-delimited format — e.g., "01:20:13:05" to
  indicate 1 day, 20 hours, 13 minutes and 5 seconds.  Time specified
  without colons will be assumed to have been given in seconds.

* "REPORT-STATE-ON-TIMEOUT[(=bool)]": directs the runtime to provide a
  detailed report on job and application process state upon job
  timeout. This option defaults to a true value if the option is given
  without a value.

* "GET-STACK-TRACES[(=bool)]": requests that the runtime provide stack
  traces on all application processes still executing upon timeout.
  This option defaults to a true value if the option is given without
  a value.

* "REPORT-CHILD-JOBS-SEPARATELY[(=bool)]": directs the runtime to
  report the exit status of any child jobs spawned by the primary job
  separately. If false, then the final exit status reported will be
  zero if the primary job and all spawned jobs exit normally, or the
  first non-zero status returned by either primary or child jobs. This
  option defaults to a true value if the option is given without a
  value.

* "AGGREGATE-HELP-MESSAGES[(=bool)]": directs the runtime to aggregate
  help messages, reporting each unique help message once accompanied
  by the number of processes that reported it. This option defaults to
  a true value if the option is given without a value.

* "FWD-ENVIRONMENT[(=bool)]": directs the runtime to forward the
  entire local environment in support of the application. This option
  defaults to a true value if the option is given without a value.

The "--rtos" command line option has no qualifiers.

Note:

  Directives are case-insensitive.  "FWD-ENVIRONMENT" is the same as
  "fwd-environment".
#
[notifications]

PMIx provides a number of event codes relevant to fault tolerance. These
can be captured by a process using the PMIx_Register_event_handler API.
Process-oriented events include:

* PMIX_ERR_PROC_RESTART - one or more processes have restarted. The IDs of
  the processes will be included in the provided info array

* PMIX_ERR_PROC_CHECKPOINT - please checkpoint the process

* PMIX_ERR_PROC_MIGRATE - one or more processes have migrated to a new
  location. The IDs of the processes will be included in the provided
  info array

* PMIX_JCTRL_PREEMPT_ALERT - the resource manager will be preempting the
  process. Additional info (e.g., delay before preemption) may be included
  in the provided info array

* PMIX_ERR_PROC_REQUESTED_ABORT - one or more processes have called
  "abort". The IDs of the processes will be included in the provided info
  array

* PMIX_ERR_PROC_TERM_WO_SYNC - one or more processes have terminated without
  calling PMIx_Finalize. The IDs of the processes will be included in the
  provided info array

* PMIX_EVENT_PROC_TERMINATED - one or more processes have normally terminated.
  The IDs of the processes will be included in the provided info array

* PMIX_ERR_PROC_KILLED_BY_CMD - one or more processes were ordered to die. The
  IDs of the processes will be included in the provided info array

* PMIX_ERR_PROC_FAILED_TO_START - one or more processes failed to start. The IDs
  of the processes will be included in the provided info array

* PMIX_ERR_PROC_ABORTED_BY_SIG - one or more processes were terminated by signal.
  The IDs of the processes will be included in the provided info array

* PMIX_ERR_PROC_SENSOR_BOUND_EXCEEDED - one or more processes exceeded their sensor
  limits (e.g., consumed too much memory) and have been terminated. The IDs of the
  processes will be included in the provided info array

* PMIX_ERR_EXIT_NONZERO_TERM - one or more processes have terminated with a non-zero
  exit status. The IDs of the processes will be included in the provided info array

Job-oriented events include:

* PMIX_EVENT_JOB_START - a new namespace has started. The namespace will be
  included in the provided info array.

* PMIX_EVENT_JOB_END - a namesdpace has terminated. The namespace will be
  included in the provided info array.


Session-oriented events include:

* PMIX_EVENT_SESSION_START - a new session has started. The session ID will be
  included in the provided info array.

* PMIX_EVENT_SESSION_END - a session has ended. The session ID will be
  included in the provided info array.

#
[runtime-options]

The "--runtime-options" command line directive is a synonym for the
"--rtos" directive.

#include#help-schizo-rtos.txt#rtos
