/* StarPU --- Runtime system for heterogeneous multicore architectures.
 *
 * Copyright (C) 2009-2023  Université de Bordeaux, CNRS (LaBRI UMR 5800), Inria
 *
 * StarPU is free software; you can redistribute it and/or modify
 * it under the terms of the GNU Lesser General Public License as published by
 * the Free Software Foundation; either version 2.1 of the License, or (at
 * your option) any later version.
 *
 * StarPU is distributed in the hope that it will be useful, but
 * WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 *
 * See the GNU Lesser General Public License in COPYING.LGPL for more details.
 */

/*! \page Scheduling Scheduling

\section TaskSchedulingPolicy Task Scheduling Policies

The basics of the scheduling policy are the following:

<ul>
<li>The scheduler gets to schedule tasks (<c>push</c> operation) when they become
ready to be executed, i.e. they are not waiting for some tags, data dependencies
or task dependencies.</li>
<li>Workers pull tasks (<c>pop</c> operation) one by one from the scheduler.
</ul>

This means scheduling policies usually contain at least one queue of tasks to
store them between the time when they become available, and the time when a
worker gets to grab them.

By default, StarPU uses the work-stealing scheduler \b lws. This is
because it provides correct load balance and locality even if the application codelets do
not have performance models. Other non-modelling scheduling policies can be
selected among the list below, thanks to the environment variable \ref
STARPU_SCHED. For instance, <c>export STARPU_SCHED=dmda</c> . Use <c>help</c> to
get the list of available schedulers.

The function starpu_sched_get_predefined_policies() returns a NULL-terminated array of all predefined scheduling policies that are available in StarPU.
Functions starpu_sched_get_sched_policy_in_ctx() and starpu_sched_get_sched_policy() return the scheduling policy of a task within a specific context or a default context, respectively.

\subsection NonPerformanceModelingPolicies Non Performance Modelling Policies

- The <b>eager</b> scheduler uses a central task queue, from which all workers draw tasks
to work on concurrently. This however does not permit to prefetch data since the scheduling
decision is taken late. If a task has a non-0 priority, it is put at the front of the queue.

- The <b>random</b> scheduler uses a queue per worker, and distributes tasks randomly according to assumed worker
overall performance.

- The <b>ws</b> (work stealing) scheduler uses a queue per worker, and schedules
a task on the worker which released it by
default. When a worker becomes idle, it steals a task from the most loaded
worker.

- The <b>lws</b> (locality work stealing) scheduler uses a queue per worker, and schedules
a task on the worker which released it by
default. When a worker becomes idle, it steals a task from neighbor workers. It
also takes priorities into account.

- The <b>prio</b> scheduler also uses a central task queue, but sorts tasks by
priority specified by the application.

- The <b>heteroprio</b> scheduler uses different priorities for the different processing units.
This scheduler must be configured to work correctly and to expect high-performance
as described in the corresponding section.

\subsection DMTaskSchedulingPolicy Performance Model-Based Task Scheduling Policies

If (<b>and only if</b>) your <b>codelets have performance models</b> (\ref
PerformanceModelExample), you should change the scheduler thanks to the
environment variable \ref STARPU_SCHED, to select one of the policies below, in
order to take advantage of StarPU's performance modelling. For instance,
<c>export STARPU_SCHED=dmda</c> . Use <c>help</c> to get the list of available
schedulers.

<b>Note:</B> Depending on the performance model type chosen, some preliminary
calibration runs may be needed for the model to converge. If the calibration
has not been done, or is insufficient yet, or if no performance model is
specified for a codelet, every task built from this codelet will be scheduled
using an <b>eager</b> fallback policy.

<b>Troubleshooting:</b> Configuring and recompiling StarPU using the \c configure option
\ref enable-verbose "--enable-verbose" displays some statistics at the end of
execution about the percentage of tasks which have been scheduled by a DM*
family policy using performance model hints. A low or zero percentage may be
the sign that performance models are not converging or that codelets do not
have performance models enabled.

- The <b>dm</b> (deque model) scheduler takes task execution performance models into account to
perform a HEFT-similar scheduling strategy: it schedules tasks where their
termination time will be minimal. The difference with HEFT is that <b>dm</b>
schedules tasks as soon as they become available, and thus in the order they
become available, without taking priorities into account.

- The <b>dmda</b> (deque model data aware) scheduler is similar to \b dm, but it also takes data transfer time
into account.

- The <b>dmdap</b> (deque model data aware prio) scheduler is similar to \b dmda,
except that it sorts tasks by priority order, which allows becoming even closer
to HEFT by respecting priorities after having made the scheduling decision (but
it still schedules tasks in the order they become available).

- The <b>dmdar</b> (deque model data aware ready) scheduler is similar to \b dmda,
but it also privileges tasks whose data buffers are already available
on the target device.

- The <b>dmdas</b> combines \b dmdap and \b dmdar: it sorts tasks by priority order,
but for a given priority it will privilege tasks whose data buffers are already
available on the target device.

- The <b>dmdasd</b> (deque model data aware sorted decision) scheduler is similar
to dmdas, except that when scheduling a task, it takes into account its priority
when computing the minimum completion time, since this task may get executed
before others, and thus the latter should be ignored.

- The <b>heft</b> (heterogeneous earliest finish time) scheduler is a deprecated
alias for <b>dmda</b>.

- The <b>pheft</b> (parallel HEFT) scheduler is similar to \b dmda, it also supports
parallel tasks (still experimental). It should not be used when several contexts using
it are being executed simultaneously.

- The <b>peager</b> (parallel eager) scheduler is similar to eager, it also
supports parallel tasks (still experimental). It should not be used when several
contexts using it are being executed simultaneously.

\subsection ExistingModularizedSchedulers Modularized Schedulers

StarPU provides a powerful way to implement schedulers, as documented in \ref
DefiningANewModularSchedulingPolicy. It is currently shipped with the following
pre-defined Modularized Schedulers :

- <b>modular-eager</b> , <b>modular-eager-prefetching</b> are eager-based Schedulers (without and with prefetching), they are
naive schedulers, which try to map a task on the first available resource
they find. The prefetching variant queues several tasks in advance to be able to
do data prefetching. This may however degrade load balancing a bit.

- <b>modular-prio</b>, <b>modular-prio-prefetching</b>, <b>modular-eager-prio</b> are prio-based Schedulers (without / with prefetching):,
similar to Eager-Based Schedulers. They can handle tasks which have a defined
priority and schedule them accordingly.
The <b>modular-eager-prio</b> variant integrates the eager and priority queue in a
single component. This allows it to do a better job at pushing tasks.

- <b>modular-random</b>, <b>modular-random-prio</b>, <b>modular-random-prefetching</b>, <b>modular-random-prio-prefetching</b> are random-based Schedulers (without/with prefetching) :
Select randomly a resource to be mapped on for each task.

- <b>modular-ws</b>) implements Work Stealing:
Maps tasks to workers in round-robin, but allows workers to steal work from other workers.

- <b>modular-heft</b>, <b>modular-heft2</b>, and <b>modular-heft-prio</b> are
HEFT Schedulers : \n
Maps tasks to workers using a heuristic very close to
Heterogeneous Earliest Finish Time.
It needs that every task submitted to StarPU have a
defined performance model (\ref PerformanceModelCalibration)
to work efficiently, but can handle tasks without a performance
model. <b>modular-heft</b> just takes tasks by order. <b>modular-heft2</b> takes
at most 5 tasks of the same priority and checks which one fits best.
<b>modular-heft-prio</b> is similar to <b>modular-heft</b>, but only decides the memory
node, not the exact worker, just pushing tasks to one central queue per memory
node. By default, they sort tasks by priorities and privilege, running first
a task which has most of its data already available on the target. These can
however be changed with \ref STARPU_SCHED_SORTED_ABOVE, \ref
STARPU_SCHED_SORTED_BELOW, and \ref STARPU_SCHED_READY .

- <b>modular-heteroprio</b> is a Heteroprio Scheduler: \n
Maps tasks to worker similarly to HEFT, but first attribute accelerated tasks to
GPUs, then not-so-accelerated tasks to CPUs.

\section TaskDistributionVsDataTransfer Task Distribution Vs Data Transfer

Distributing tasks to balance the load induces data transfer penalty. StarPU
thus needs to find a balance between both. The target function that the
scheduler \b dmda of StarPU
tries to minimize is <c>alpha * T_execution + beta * T_data_transfer</c>, where
<c>T_execution</c> is the estimated execution time of the codelet (usually
accurate), and <c>T_data_transfer</c> is the estimated data transfer time. The
latter is estimated based on bus calibration before execution start,
i.e. with an idle machine, thus without contention. You can force bus
re-calibration by running the tool <c>starpu_calibrate_bus</c>. The
beta parameter defaults to <c>1</c>, but it can be worth trying to tweak it
by using <c>export STARPU_SCHED_BETA=2</c> (\ref STARPU_SCHED_BETA) for instance, since during
real application execution, contention makes transfer times bigger.
This is of course imprecise, but in practice, a rough estimation
already gives the good results that a precise estimation would give.

*/
