=head1 NAME

oarsub - OAR job submission command.

=head1 SYNOPSIS

B<oarsub> [OPTIONS] <command>

B<oarsub> [OPTIONS] -I

B<oarsub> [OPTIONS] -r "<START DATE|now>[, <END DATE>]" [<command>]

B<oarsub> [OPTIONS] -C [<JOB ID>]

=head1 DESCRIPTION

One uses B<oarsub> to submit a job to the B<OAR resources and jobs management system>.
A job is defined by a command to run and optionally the description of a set of wanted resources to execute a task.
The I<command> can be any executable program or script file, or a script command line.

The OAR scheduler is in charge of providing a set of resources which matches the B<oarsub> resource request (see the I<-l> option). Once scheduled and launched, the job executes the command on the first node (machine) of the assigned resources, with some environment variables set (e.g. the assigned resources list, the job duration a.k.a walltime, ...). If many nodes (machines) are assigned to the job, it is let to the user's command to dispatch an execution on all nodes.

OAR defines two kinds of jobs: I<batch jobs> and I<advance reservation jobs>.

=head2 BATCH JOBS

A batch job lets the OAR scheduler decide when to execute the job, depending on when the requested resources are available. If not using the I<-I> option, B<oarsub> returns immediately: the job is queued (use the B<oarstat> command to list the queued jobs). When using the B<-I> option, B<oarsub> waits until the job can start then opens a interactive shell on the first of the assigned nodes.

=head2 ADVANCE RESERVATION JOBS

Calling B<oarsub> with the I<-r> option creates an advance reservation job instead of a batch job. An advance reservation job lets the user decide when the job should start (and optionally end). B<oarsub> waits for a validation of the resources availability. Whether a set of resources matching the resource request of the job is available, an I<OK> or I<KO> message is returned. I<OK> means that the job will run at the requested date. I<KO> means that the job is refused.

=head1 OPTIONS

=over

=item B<-I, --interactive>

Request an interactive job.

=item B<-C, --connect> [<JOB ID>]

Connect to a running job. Job id may be omitted if only one job is running.

=item B<-l, --resource> <DESCRIPTION>[,<WALLTIME>]

Set the I<description> of the requested resources for the job and a I<walltime> which specifies an estimation of the maximum duration of the job (the job process can terminate earlier). 
The walltime format is I<hours>:I<minutes>:I<seconds> (seconds and minutes can be omitted). 

  oarsub -l /host=4/cpu=1,walltime=2:00:00

Multiple I<-l> options can be given to define a moldable job: a job accepting different combination of resource definitions. Example:

  oarsub -l cpu=2,walltime=20:00:00 -l cpu=4,walltime=10:00:00 -l cpu=8,walltime=5:00:00 ./script.sh

OAR will allocate resources to the job accordingly to one of the 3 given definitions depending of the load of the cluster, and preferring earliest possible start.

One can also request different groups of resources, for example:

  oarsub -l "{mem > 64}/host=1+{mem < 48}/host=3",walltime=1:00:00 -I

The job requests to run on 1 host with the property "mem" > 64 and 3 hosts with property "mem" < 48.
The syntax between braces, {...}, is the same as the one used for I<-p> option.

=item B<--array> <NUMBER>

Submit an array job containing "NUMBER" subjobs. All the subjobs share the same array_id but each subjob is independent and has its own job_id. All the subjobs have the same characteristics (script, requirements) and can be identified by an environment variable $OAR_ARRAY_INDEX.

Array jobs can neither be Interactive (-I) nor a reservation (-r).


=item B<--array-param-file> <FILE>

Submit a parametric array job. Each non-empty line of "FILE" defines the parameters for a subjob. All the subjobs have the same characteristics (script, requirements) and can be identified by an environment variable $OAR_ARRAY_INDEX.   '#' is the comment sign.

Parameters are given as arguments to the command.

Parametric array jobs can neither be Interactive (-I) nor a reservation (-r).

=item B<-S, --scanscript>

Request I<oarsub> to scan the script file given as the job command, for extra options to apply to the job (lines starting with #OAR)

=item B<-q, --queue> <QUEUE>

Set the queue to submit the job to.

=item B<-p, --property> "<SQL WHERE CLAUSE>"

Add constraints to the wanted resource for the job, using the resources properties.
Constraints use the SQL WHERE CLAUSE syntax.

=item B<-r, --reservation> "<START DATE|now>[, <END DATE>]"

Request that the job starts (and optionally ends) at a specified time. Using
this option, the job is called an I<advance reservation>.

Start and end dates use the I<YYYY-MM-DD hh:mm:ss> format. If I<YYYY-MM-DD> is
omitted, it defaults to the current day. In I<hh:mm:ss>, I<ss> and I<mm> can
be omitted. Quotes are only required when spaces are present.  When an end date
is provided the job I<walltime> is inferred, unless provided in the I<-l>
resources request which prevails. The special keyword B<now> can be used as the
start date to request an advance reservation job that starts just now.

=item B<--checkpoint> <DELAY>

Enable the checkpointing mechanism for the job.
A signal will be sent DELAY seconds before the walltime to the first processus of the job (on the first node of the job resources).

=item B<--signal> <#SIG>

Specify the signal to used to trigger the checkpointing. Use signal numbers (see kill -l), default is I<12> (SIGUSR2).

=item B<-t, --type> <TYPE>

Specify a specific type (I<besteffort>, I<timesharing>, I<idempotent>, I<cosystem>, I<deploy>, I<noop>, I<container>, I<inner>, I<token:xxx=yy>,... )

Notes:
  - a job with the B<besteffort> type will be scheduled with the lowest priority and will be killed if a "normal" job needs its resources.
  - a job with the B<idempotent> type will be automatically resubmitted if its exit code is 99 and its duration > 60s.
  - a job with the B<idempotent> and B<besteffort> types but with no B<checkpoint> delay will automatically be resubmitted whenever killed by OAR before its normal termination to execute a non-besteffort jobs.
  - a job with the B<idempotent> and B<besteffort> types with a B<checkpoint> delay will be automatically resubmitted if its exit code is 99 and its duration > 60s (that is, as if only B<idempotent> was present).
  - a job with the B<noop> type does nothing except reserving the resources. It is ended at the end of it walltime or when using the oardel command.

The I<-t> option can be used several times to provide different job types to the job.

=item B<-d, --directory> <DIR>

Specify the directory where to launch the command (default is current directory)

=item B<--project> <TXT>

Specify a name of a project the job belongs to.

=item B<-n, --name> <TXT>

Specify an arbitrary name for the job.

=item B<-a, --anterior> <OAR JOB ID>

Previously submitted job that the new job execution must depend on. The new job will only start upon the end of the previous one.

=item B<--notify> <TXT>

Specify a notification method (email or command to execute).
Ex: 
 --notify "mail:name@domain.com"
 --notify "exec:/path/to/script args"

Arguments are job_id, job_name, TAG, comment

TAG can be:
  - RUNNING: when the job is launched
  - END: when the job is finished normally
  - ERROR: when the job is finished abnormally
  - INFO: used when oardel is called on the job
  - SUSPENDED: when the job is suspended
  - RESUMING: when the job is resumed

By default all TAGs are triggered. It is possible to specify which TAGs must be triggered.
Ex:
 --notify "[END,ERROR]mail:name@domain.com"
 --notify "[RUNNING]mail:name@domain.com"
 --notify "[RUNNING,END,ERROR]exec:/path/to/script args"

=item B<--resubmit> <OAR JOB ID>

Resubmit the given job as a new one.

=item B<-k, --use-job-key>

Activate the job-key mechanism: a job-key will be generated for the job, which can be used my the B<oarsh> command in place of the B<OAR_JOB_ID>. That job-key also allows one to connect to a job from a machine which is outside the OAR cluster the job belong to (e.g. in a grid of OAR clusters), given the job-key is available on that machine.

The job-key mechanism may be activated by default in the configuration of your OAR cluster. In that case, the -k option is useless. 

Please note that this option is mostly useful when used together with the -e option or the -i option (see below).

=item B<-e, --export-job-key-to-file>

Export the job-key to a file (the %jobid% pattern is automatically replaced in the filename). I<Warning>: the file will be overwritten, whenever it already exists. 

=item B<-i, --import-job-key-from-file>

Import the job-key to use (you may reuse an exported job-key from a previous job for instance) from a file, instead of generating a new one. One may also use the B<OAR_JOB_KEY_FILE> environment variable to set the job-key file.

=item B<--import-job-key-inline>

Import the job-key to use inline (as text in the command line), instead of generating a new one.

=item B<-O> <FILE>

Specify the file that will store the standard output stream of the job. The %jobid% and %jobname% patterns are automatically replaced.

=item B<-E> <FILE>

Specify the file that will store the standard error stream of the job. The %jobid% and %jobname% patterns are automatically replaced.

=item B<--hold>

Set the job state into Hold instead of Waiting, so that it is not scheduled as long as not resumed (the I<oarresume> command allows one to turn it back into the Waiting state).

=item B<-D, --dumper>

Print the command results in Perl's Data::Dumper format.

=item B<-X, --xml>

Print the command results in the XML format.

=item B<-Y, --yaml>

Print the command results in the YAML format.

=item B<-J, --json>

Print the command results in the JSON format.

=item B<-h, --help>

Print this help message.

=item B<-V, --version>

Print the version of OAR.

=item B<-v>

Verbose mode, oarsub and admissions rules (if compatible) will print more information on terminal. Multiple -v can be used to increase the level of verbosity (max: 2).

=back

=head1 ENVIRONMENT

=over

=item B<OAR_FILE_NODES> or B<OAR_NODE_FILE> or B<OAR_NODEFILE>

Pathname to the file containing the list of the nodes that are assigned to the job. 

=item B<OAR_JOB_NAME>

Name of the job as given using the B<-n> option.

=item B<OAR_JOB_ID> or B<OAR_JOBID>

Id of the job. Each job get a unique job identifier. This identifier can be use to retrieve information about the job using B<oarstat>, or to connect to a running job using B<oarsub -C> or B<oarsh> for instance.

=item B<OAR_ARRAY_ID> or B<OAR_ARRAYID>

Array Id of the job. Every array job gets an unique array identifier that is shared by all the subjobs of the array job. This identifier can be used to identify the different subjobs pertaining to a same array job. Array Id can also be used to deal with all the subjobs of a given array at once (by means of the --array  option in the case of oarstat, oarhold, oarresume and oardel). NB: regular jobs are array jobs with only one subjob.
 

=item B<OAR_JOB_INDEX> or B<OAR_JOBINDEX>

Array Index of the job: within an array job, each subjob gets a unique (for a given array) index, starting from 0, which can be used to identify the subjob.

=item B<OAR_JOB_WALLTIME> and B<OAR_JOB_WALLTIME_SECONDS>

Walltime of the job in the hh:mm:ss format and in seconds.

=item B<OAR_RESOURCE_PROPERTIES_FILE>

Pathname to the file containing the list of all resources assigned for the job, and their value. See I<oarprint> also.

=item B<OAR_PROJECT_NAME>

Name of the project the job is part of, as given using the B<--project> option.

=item B<OAR_STDOUT> and B<OAR_STDERR>

Pathname to the files storing the standard output and standard error of the job executable, if not running in interactive mode.

=item B<OAR_WORKING_DIRECTORY> aka B<OAR_WORKDIR> aka B<OAR_O_WORKDIR>

Working directory for the job. The job executable will be executed in that directory, on the first node which is allocated to the job.

=item B<OAR_JOB_KEY_FILE>

Key file to use for the submission (or I<oarsh>) if using the job-key mechanism (-k or --use-job-key option). One may provide the job-key to import using the -i or --import-job-key-from-file option as well.

=back

=head1 SCRIPT

When submitting a job using a script file, that script file can contain some extra OAR options, with lines starting with B<#OAR> and using the same option syntax as described above.

=head1 EXAMPLES

=head2 Job submission with arguments

=over

 oarsub -l /nodes=4 -I

 oarsub -q default -l /nodes=10/cpu=3,walltime=50:30:00 -p "switch = 'sw1'" /home/username/path/to/my/prog

 oarsub -r "2009-04-27 11:00:00" -l /nodes=12/cpu=2

 oarsub -C 154

=back

=head2 Array job submission with 10 identical subjobs

=over

 oarsub -l /nodes=4 /home/usename/path/to/my/prog --array 10

=back

=head2 Parametric array job submission

=over

 oarsub /home/users/toto/prog --array-param-file /home/username/path/to/params.txt

=back

With /home/username/path/to/param.txt containing for instance:

 # my param file
 # a subjob with a single parameter
 p100
 # a subjob without parameter
 ""
 # a subjob with two strings as parameters
 "arg1a arg1b arg1c" "arg2a arg2b"

=head2 Job submission with a job script

=over

 oarsub -S /home/username/path/to/my/script.sh

=back

With /home/username/path/to/my/script.sh containing for instance:

 #!/bin/bash
 #OAR -l /nodes=4/cpu=1,walltime=3:15:00
 #OAR -p switch = 'sw3' or switch = 'sw5'
 #OAR -t besteffort
 #OAR -t type2
 #OAR -k
 #OAR -e /path/to/job/key
 #OAR --stdout stdoutfile.log
 /home/username/path/to/my/prog


=head1 SEE ALSO

oarprint(1), oarsh(1), oardel(1), oarstat(1), oarnodes(1), oarhold(1), oarresume(1)

=head1 COPYRIGHTS

 Copyright 2003-2021 Laboratoire d'Informatique de Grenoble (http://www.liglab.fr). This software is licensed under the GNU General Public License Version 2 or above. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

