
=encoding utf8

=head1 TITLE

DRAFT: Synopsis 17: Processes and Concurrency

=head1 AUTHORS

    Elizabeth Mattijsen <liz@dijkmat.nl>
    Audrey Tang <audreyt@audreyt.org>
    Christoph Buchetmann
    Tim Nelson <wayland@wayland.id.au>

=head1 VERSION

    Created: 13 Jun 2005

    Last Modified: 27 Feb 2009
    Version: 4

This draft document is a paste together from various sources.  The bulk of it is simply
the old S17-concurrency.pod, which dealt only with concurrency.  Signals were added from
S16-io, but haven't been merged with the conflicting S17 signals doco.  An event loop
section has been added here because a) Larry mentioned the idea, and b) Moritz suggested
that http://www.seas.upenn.edu/~lipeng/homepage/unify.html be our model for concurrency,
and in that model, an event loop underlies the threads.

=head1 Overview

An event loop underlies everything in this document.  POSIX signals can interact with
this, and concurrency is built on top of it.  Naturally, IPC (inter-process communication)
is documented here too (XXX or should be :) ).

=head1 The Event Loop

=head1 POSIX Signals

The %*SIG variable contains a Hash of Proc::Signals::Signal.

    class Proc::Signals::Signal {
        has $exception; # This specifies what exception will be raised when this signal is received
        has $interrupt; # See siginterrupt(3)
        has $blocked;   # Is this signal blocked?  cf. sigprocmask
    }

The @*SIGQUEUE array contains a queue of the signals that are blocked and queued.

The standard POSIX signals simply raise control exceptions that are handled as normal
through the control signal handler, and caught by CONTROL blocks, as specified in S04.

To declare your main program catches INT signals, put a CONTROL block anywhere
in the toplevel to handle exceptions like this:

 CONTROL {
     when Error::Signal::INT { ... }
 }

=head2 Signal defaults

The signals have defaults as specified in the table below.  $blocked always defaults to
false.

 Signal     Default Exception
 ------     -----------------
 SIGHUP     ControlExceptionSigHUP
 SIGINT     ControlExceptionSigINT
 SIGQUIT    ControlExceptionSigQUIT
 SIGILL     ControlExceptionSigILL
 SIGABRT    ControlExceptionSigABRT
 SIGFPE     ControlExceptionSigFPE
 SIGKILL    ControlExceptionSigKILL
 SIGSEGV    ControlExceptionSigSEGV
 SIGPIPE    ControlExceptionSigPIPE
 SIGALRM    ControlExceptionSigALRM
 SIGTERM    ControlExceptionSigTERM
 SIGUSR1    ControlExceptionSigUSR1
 SIGUSR2    ControlExceptionSigUSR2
 SIGCHLD    ControlExceptionSigCHLD
 SIGCONT    ControlExceptionSigCONT
 SIGSTOP    ControlExceptionSigSTOP
 SIGTSTP    ControlExceptionSigTSTP
 SIGTTIN    ControlExceptionSigTTIN
 SIGTTOU    ControlExceptionSigTTOU
 SIGBUS     ControlExceptionSigBUS
 SIGPROF    ControlExceptionSigPROF
 SIGSYS     ControlExceptionSigSYS
 SIGTRAP    ControlExceptionSigTRAP
 SIGURG     Undefined
 SIGVTALRM  ControlExceptionSigVTALRM
 SIGXCPU    ControlExceptionSigXCPU
 SIGXFSZ    ControlExceptionSigXFSZ
 SIGEMT     ControlExceptionSigEMT
 SIGSTKFLT  ControlExceptionSigSTKFLT
 SIGIO      ControlExceptionSigIO
 SIGPWR     ControlExceptionSigPWR
 SIGLOST    ControlExceptionSigLOST
 SIGWINCH   Undefined

=head2 Signal exceptions

A table below describes the exceptions.

Each of these has a default action as well.  The possible actions are:

 Term   Default action is to terminate the process.

 Ign    Default action is to ignore the signal ($signal.exception is undef by default)

 Core   Default action is to terminate the process and dump core (see core(5)).

 Stop   Default action is to stop the process.

 Cont   Default action is to continue the process if it is currently stopped.

Some actions do the Resumeable role.  An exception listed in the table below that does the
Resumeable role is marked with a * in the R column.

The exceptions are:

 Signal                   Action   R Comment
 ----------------------------------------------------------------------
 ControlExceptionSigHUP    Term    ? Hangup detected on controlling terminal or death of controlling process
 ControlExceptionSigINT    Term    ? Interrupt from keyboard
 ControlExceptionSigQUIT   Core    ? Quit from keyboard
 ControlExceptionSigILL    Core    ? Illegal Instruction
 ControlExceptionSigABRT   Core    ? Abort signal from abort(3)
 ControlExceptionSigFPE    Core    ? Floating point exception
 ControlExceptionSigKILL   Term    ? Kill signal
 ControlExceptionSigSEGV   Core      Invalid memory reference
 ControlExceptionSigPIPE   Term    ? Broken pipe: write to pipe with no readers
 ControlExceptionSigALRM   Term    ? Timer signal from alarm(2)
 ControlExceptionSigTERM   Term    ? Termination signal
 ControlExceptionSigUSR1   Term    ? User-defined signal 1
 ControlExceptionSigUSR2   Term    ? User-defined signal 2
 ControlExceptionSigCHLD   Ign     * Child stopped or terminated
 ControlExceptionSigCONT   Cont    * Continue if stopped
 ControlExceptionSigSTOP   Stop    ? Stop process
 ControlExceptionSigTSTP   Stop    ? Stop typed at tty
 ControlExceptionSigTTIN   Stop    ? tty input for background process
 ControlExceptionSigTTOU   Stop    ? tty output for background process
 ControlExceptionSigBUS    Core    ? Bus error (bad memory access)
 ControlExceptionSigPROF   Term    ? Profiling timer expired
 ControlExceptionSigSYS    Core    ? Bad argument to routine (SVr4)
 ControlExceptionSigTRAP   Core    ? Trace/breakpoint trap
 ControlExceptionSigURG    Ign     ? Urgent condition on socket (4.2BSD)
 ControlExceptionSigVTALRM Term    ? Virtual alarm clock (4.2BSD)
 ControlExceptionSigXCPU   Core    ? CPU time limit exceeded (4.2BSD)
 ControlExceptionSigXFSZ   Core    ? File size limit exceeded (4.2BSD)
 ControlExceptionSigEMT    Term    ?
 ControlExceptionSigSTKFLT Term    ? Stack fault on coprocessor (unused)
 ControlExceptionSigIO     Term    ? I/O now possible (4.2BSD)
 ControlExceptionSigPWR    Term    ? Power failure (System V)
 ControlExceptionSigLOST   Term    ? File lock lost
 ControlExceptionSigWINCH  Ign     ? Window resize signal (4.3BSD, Sun)

See L<S04-control> for details on how to handle exceptions.

XXX I'm unsure how the actions in the table above can be made to make sense.  The Ign
actions are already dealt with because %SIG{CHLD}.exception already defaults to undef.
The Term action will probably be self-solving (ie. will terminate the process).  The
others I'm just plain unsure about.  XXX

=head1 Alarm

XXX Everything about Alarm is from the old S17-concurrency.pod

An alarm is just a pre-arranged exception to be delivered to your program.

By the time alarm has arrived, the current block may have already finished
executing, so you would need to set up CATCH blocks in places where an alarm
can rise to handle it properly.

You can request an alarm using the number of seconds, or with a target date.
It returns a proxy alarm object that you can do interesting things with.

    multi Alarm *alarm (Num $seconds = $CALLER::_, &do = {die Sig::ALARM}, :$repeat = 1)
    multi Alarm *alarm (Date $date, &do = {die Sig::ALARM}, :$repeat = 1)

Perl 6's C<alarm> has three additional features over traditional alarms:

=head2 Multiple and Lexical Alarms

One can set up multiple alarms using repeated alarm calls:

    {
        my $a1 = alarm(2);
        my $a2 = alarm(2);
        sleep 10;
        CATCH {
            is critical; # if you don't want $a2 to be raised inside this
            when Sig::ALARM { ... }
        }
    }

To stop an alarm, call C<$alarm.stop>.  The C<alarms> method for Conc objects
(including process and threads) returns a list of alarms currently scheduled
for that concurrent context.

When an alarm object is garbage collected, the alarm is stopped automatically.
Under void context, the implicit alarm object can only be stopped by querying
C<.alarms> on the current process.

We are not sure what C<alarm(0)> would mean.  Probably a deprecation warning?

=head2 Repeated Alarms

If you request a repeated alarm using the C<repeated> named argument, it will
attempt to fire off the alarm that many times.  However, the alarm will be
suppressed when inside a C<CATCH> block that's already handling the exception
raised by I<same> alarm.

To repeat 0 times is to not fire off any alarms at all.  To repeat +Inf times
is to repeat over and over again.

=head2 Callbacks in Alarms

You can arrange a callback (like JavaScript's setTimeOut) in C<alarm>, which
will then be invoked with the then-current code as caller.

If you set up such a callback to another Conc object, what happens is just like
when you called C<.die> on behalf of that object -- namely, the callback
closure, along with anything it referenced, is shared to the target Conc
context.

Unlike in Perl 5's ithreads where you cannot share anything after the fact,
this allows passing shared objects in an C<ad-hoc> fashion across concurrent
parts of the program.  Under the default (multiplexing) concurrency model, this
is basically a no-op.

=head1 IPC

=head1 Concurrency

This is a draft document. After being some time under the surface of Perl 6
development this is a attempt to document working concurrency issues, list the
remaining todos and mark the probably obsolete and redundant points.

=head2 Overview

Concurrency can take many forms in Perl 6.  With varying degrees of
explicitness and control capabilities.  This document attempts to describe
what these capabilities are and in which form they can be accessed in Perl 6.

=head3 Processes, threads, fibers?

Concurrency comes in many shapes and forms.  Most Perl users are used to
the concept of a "process" or a "thread" (usually depending on the OS they
work on).  Some systems even are familiar with very lightweight threads
called "fibers".

When discussing issues about concurrency with different people, it soon
becomes apparent that everybody has his own set of "understandings" about
what each word means, which doesn't make it any easier to describe
Perl 6 concurrency.

It seemed the most natural to use the word "thread" to describe a process
which has its own context, but also shares context with 0 or more
concurrently running processes.  Depending on your OS, or even specific
version of your OS, this could still be a single "process" from the OS's
point of view.  Or it could contain an OS process for each thread.  Or
any mixture of these two implementations.

In this document we try to be agnostic about this: all we know in Perl 6
are "threads", which have their own context and share context with other
concurrently running "threads".  Whether they be process, threads or fibres
at the OS level should not matter at the Perl 6 level.

And for sake of consistency, an unthreaded "normal" program is considered
to be also running in a single thread.

=head3 Variables

In the past, there have been two models for concurrent processes in Perl.
In general, these are referred to as "5.005 threads" (C<perldoc perlothrtut>)
and "ithreads" (C<perldoc perlthrtut>).

The main difference between these two models from a programmer's point of
view, is that variables in "5.005 threads" are shared by default.  Whereas
in the "ithreads" model, only variables that have been indicated to be
"shared", are actually shared between threads.  All other variable values
are actually copies of the variable's value in the "parent" thread.

With regards to variables, the concurrency model of Perl 6 is closer to the
"5.005 threads" model than it is to the "ithreads" model.  In fact, all
variables "visible" to a particular scope in Perl 6 will be accessible and
modifiable (if allowed to do so) from all of the concurrent processes that
start from that scope.  In that sense, one could consider the "ithreads"
model as a historical diversion: the Perl 6 concurrency picks up where the
"5.005 threads" path left off.

(EM: maybe point out that the "ithreads" behaviour can be simulated with
some kind of copy-on-write magic to be automagically added to all variable
access inside a thread, except for those with an explicit "is shared"
attribute?)

=head2 Contend/Maybe/Defer

=head3 No user accessible locks

Differently from any current concurrent process implementation in Perl, there
are no user accessible locks.  Instead, the concept of Software
Transactional Memory is used.  This is in concept similar to the use of

 BEGIN TRANSACTION
 ... do your uninterruptible actions
 COMMIT

in the database world.  More interestingly, this also includes the concept of
rollback:

 BEGIN TRANSACTION
 ... do your stuff, but impossible to complete: ROLLBACK

This causes the state of the process to be reverted to the state at the moment
the BEGIN TRANSACTION was executed.

Perl 6 supports this concept through C<contend> blocks.

These sections are guaranteed to either be completed totally (when the Code
block is exited), or have their state reverted to the state at the start of
the Code block (with the L<defer> statement).

(EM: maybe point out if / how old style locks can be "simulated", for those
needing a migration path?)

=head3 Atomic Code blocks

    my ($x, $y);
    sub c {
        $x -= 3; $y += 3;
        $x < 10 or defer;
    }
    sub d {
        $x += 3; $y -= 3;
        $y < 10 or defer;
    }

    contend {
        # ...
        maybe { c() } maybe { d() };
        # ...
    }

A Code block can be prefixed with C<contend>.  This means that code executed
inside that scope is guaranteed not to be interrupted in any way.

The start of a block marked C<contend> also becomes a I<checkpoint> to
which execution can return (in exactly the same state) if a problem occurs
(a.k.a. a L<defer> is done) inside the scope of the Code block.

=head4 defer

The C<defer> function basically restores the state of the thread at the
last checkpoint and will wait there until an external event allows it to
potentially run that atomic C<contend> section of code again without having
to defer again.

If there are no external events possible that could restart execution, an
exception will be raised.

The last checkpoint is either the outermost C<contend> boundary, or
the most immediate caller constructed with C<maybe>.

=head4 maybe

The C<maybe> statement causes a checkpoint to be made for C<defer> for
each block in the C<maybe> chain, creating an alternate execution path to
be followed when a C<defer> is done.  For example:

    maybe {
        ...
        some_condition() or defer;
        ...
    } maybe {
        ...
        some_other_condition() or defer;
        ...
    } maybe {
        ...
    }

If placed outside a C<contend> block, the C<maybe> statement creates its
own C<contend> barrier.

=head4 limitations

Because Perl 6 must be able to revert its state to the state it had at the
checkpoint, it is not allowed to perform any non-revertible actions.  These
would include reading / writing from file handles that do not support
C<seek> (such as sockets).  Attempting to do so will cause a fatal error
to occur.

This will probably need to be expanded to all objects: any object that has
some interface with data "outside" of the knowledge of the language (e.g. an
interface with an external XML library) would also need to provide some
method for freezing a state, and restoring to a previously frozen state.

If you're not interested in revertability, but are interested in
uninterruptability, you could use the "is critical" trait.

=head4 Critical Code blocks

 sub tricky is critical {
     # code accessing external info, not to be interrupted
 }

 if ($update) {
     is critical;
     # code accessing external info, not to be interrupted
 }

A Code block marked "is critical" can not be interrupted in any way.  But
since it is able to access non-revertible data structures (such as
non-seekable file handles), it cannot do a C<defer> as it would be impossible
to restore the state to the beginning of the Code block.

=head4 Mixing Atomic and Critical

Both "atomic" as well as "critical" propagate down the call chain.  This means
that any subroutine that in itself is not "atomic" or "critical" becomes
uninterruptible if called inside a code block that is marked as "atomic" or
"critical".

Atomic Code blocks called inside the call chain of a "critical" code block
do not pose a problem, as they are more restrictive.

Any code that attempts to perform any non-revertible action (e.g. reading
from a socket) will cause a fatal error when called inside the call chain
of an Atomic Code block.

=head2 Coroutines

Coroutines are covered in S07

=head2 Threads

All outside of a thread defined variables are
shared and transactional variables by default

Program will wait for _all_ threads.
Unjoined threads will be joined at the beginning of the END
block batch of the parent thread that spawned them

=head3 Thread creation

A thread will be created using the keyword C<async> followed by
a codeblock being executed in this thread.

   my $thr = async {
       ...do something...
       END { }
   };


=head3 Thread status and attributes

=over

=item Self reflection

TODO: how you can access thread attributes inside a thread

   async {
      say "my tid is ", +self;
   };

=item started

start time

=item finished

end time

=item waiting

suspended (not diff from block on wakeup signal)
waiting on a handle, a condition, a lock, et cetera
otherwise returns false for running threads
if it's finished then it's undef(?)

=item current_continuation

the CC currently running in that thread

=item wake_on_readable, wake_on_writable, wake_on

TODO: IO objects and containers gets concurrency love!

 $obj.wake_on_either_readable_or_writable_or_passed_time(3); # fixme fixme
 $obj.wake_on:{.readable} # busy wait, probably

 my @a is Array::Chan = 1..Inf;
 async { @a.push(1) };
 async { @a.blocking_shift({ ... }) };
 async { @a.unshift({ ... }) };

=back

=head3 Thread operators

=over

=item Stringify

Stringify to something sensible (eg. "<Conc:tid=5>");

  my $thr = async { ... };
  say ~$thr;

=item Numerify

Numify to TIDs (as in pugs)

  my $thr = async { ... };
  say +$thr;

=item Enumerable

TODO: Enumerable with Conc.list

=back

=head3 Thread methods

=over

=item yield

TODO: Conc.yield (if this is to live but deprecated, maybe call it sleep(0)?)

=item sleep

sleep() always respects other threads, thank you very much

=item join

wait for invocant to finish (always item cxt)

  my $thr = async { ... };
  $thr.join();

=item die

throw exception in the invocant thread

=item alarm

set up alarms

=item alarms

query existing alarms

=item suspend

pause a thread; fail if already paused

=item resume

revive a thread; fail if already running

=item detach

survives parent thread demise (promoted to process)
process-local changes no longer affects parent
tentatively, the control methods still applies to it
including wait (which will always return undef)
also needs to discard any atomicity context

=item "is throttled" trait

TODO:

    method throttled::trait_auxiliary:<is> ($limit=1, :$key=gensym()) {
        # "is throttled" limits max connection to this Code object
        # the throttling is shared among closures with the same key
        # the limit may differ on closures with the same key.
        # if the counter with the "key" equals or exceeds a closure's limit,
        # the closure can't be entered until it's released
        # (this can be trivially implemented using contend+defer)
    }

    class Foo {
        method a is throttled(:limit(3) :key<blah>) { ... }
        method b is throttled(:limit(2) :key<blah>) { ... }
    }
    my Foo $f .= new;
    async { $f.a }
    async { $f.b }

=back


=head3 Continuations

TODO document

=head3 Junctive Autothreading and Hyper Operations

Live in userland for the time being.


=head3 Still more or less unorganized stuff


    ### INTERFACE BARRIER ###
    module Blah;
    {

        is atomic;   # contend/maybe/whatever other rollback stuff
                     # limitation: no external IO (without lethal warnings anyway)
                     # can't do anything irreversible

        is critical; # free to do anything irreversible
                     # means "don't interrupt me"
                     # in system with critical section, no interrupts from
                     # other threads will happen during execution
                     # you can't suspend me

        my $boo is export;
        $boo = 1;

        # We decree that this part forms the static interface
        # it's run once during initial compilation under the
        # Separate Compilation doctrine and the syms sealed off
        # to form part of bytecode syms headers
        %CALLER::<&blah> = { 1 }; # work - adds to export set
        die "Eureka!" if %CALLER::<$sym>; # never dies

        # BEGIN { $boo = time };

        sub IMPORT {
            # VERY DYNAMIC!

            our $i = time;
            %CALLER::<&blah> = { 1 }; # work - adds to export set
            die "Eureka!" if %CALLER::<$sym>; # probes interactively
        }
    }
    ### INTERFACE BARRIER ###

=head3 See also

=over

=item *

L<S16/IE<sol>O Considerations>

=item *

L<S16/File Descriptors>

=item *

L<S16/Sockets>

=back

=cut

=head1 Additions

Please post errors and feedback to perl6-language.  If you are making
a general laundry list, please separate messages by topic.

=cut

=for vim:set expandtab sw=4:
