
\chapter{ Lwt manual }
\label{lwt:manual}

\section{ Introduction }

When writing a program, a common developer's task is to handle IO
operations. Indeed most software interact with several different
resources, such as:



\begin{itemize}
\item  the kernel, by doing system calls
\item  the user, by reading the keyboard, the mouse, or any input device
\item  a graphical server, to build graphical user interface
\item  other computers, by using the network
\item  ...

\end{itemize}

When this list contains only one item, it is pretty easy to
handle. However as this list grows it becomes harder and harder to
make everything works together. Several choices have been proposed
to solve this problem:



\begin{itemize}
\item  using a main loop, and integrate all components we are
interacting with into this main loop.
\item  using preemptive system threads

\end{itemize}

Both solutions have their advantages and their drawbacks. For the
first one, it may work, but it becomes very complicated to write
a piece of asynchronous sequential code. The typical example is
graphical user interfaces freezing and not redrawing themselves
because they are waiting for some blocking part of the code to
complete.



If you already wrote code using preemptive threads, you should know
that doing it right with threads is a hard job. Moreover system
threads consume non negligible resources, and so you can only launch
a limited number of threads at the same time. Thus this is not a
real solution.



{\tt Lwt} offers a new alternative. It provides very light-weight
cooperative threads; ``launching'' a thread is a very fast
operation, it does not require a new stack, a new process, or
anything else. Moreover context switches are very fast. In fact, it
is so easy that we will launch a thread for every system call. And
composing cooperative threads will allow us to write highly
asynchronous programs.



In a first part, we will explain the concepts of {\tt Lwt}, then we will
describe the many sub-libraries of {\tt Lwt}.



\section{ The Lwt core library }

In this section we describe the basics of {\tt Lwt}. It is advised to
start an ocaml toplevel and try the given code examples. To start,
launch {\tt ocaml} in a terminal or in emacs with the tuareg
mode, and type:



\begin{verbatim}
# #use "topfind";;
# #require "lwt.simple-top";;
\end{verbatim}
\medskip

\noindent
{\tt lwt.simple-top} makes sure {\tt Lwt} threads can run while
using the toplevel. You do not need it if your are using {\tt utop}.



\subsection{ Lwt concepts }

Let's take a classical function of the {\tt Pervasives} module:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# Pervasives.input_char;
- : in_channel -> char = <fun>
\end{lstlisting}
\medskip

\noindent
This function will wait for a character to come on the given input
channel, and then return it. The problem with this function is that it is
blocking: while it is being executed, the whole program will be
blocked, and other events will not be handled until it returns.



Now let's look at the lwt equivalent:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# Lwt_io.read_char;;
- : Lwt_io.input_channel -> char Lwt.t = <fun>
\end{lstlisting}
\medskip

\noindent
As you can see, it does not return a character but something of
type {\tt char Lwt.t}. The type {\tt 'a Lwt.t} is the type
of threads returning a value of type {\tt 'a}. Actually the
{\tt Lwt\_io.read\_char} will try to read a character from the
given input channel and \emph{immediatly} returns a light-weight
thread.



Now, let's see what we can do with a {\tt Lwt} thread. The following
code creates a pipe, and launches a thread reading on the input side:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# let ic, oc = Lwt_io.pipe ();;
val ic : Lwt_io.input_channel = <abstr>
val oc : Lwt_io.output_channel = <abstr>
# let t = Lwt_io.read_char ic;;
val t : char Lwt.t = <abstr>
\end{lstlisting}
\medskip

\noindent
We can now look at the state of our newly created thread:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# Lwt.state t;;
- : char Lwt.state = Sleep
\end{lstlisting}
\medskip

\noindent
A thread may be in one of the following states:



\begin{itemize}
\item  {\tt Return x}, which means that the thread has terminated
successfully and returned the value {\tt x}
\item  {\tt Fail exn}, which means that the thread has terminated,
but instead of returning a value, it failed with the exception
{\tt exn}
\item  {\tt Sleep}, which means that the thread is currently
sleeping and has not yet returned a value or an exception

\end{itemize}

The thread {\tt t} is sleeping because there is currently nothing
to read from the pipe. Let's write something:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# Lwt_io.write_char oc 'a';;
- : unit Lwt.t = <abstr>
# Lwt.state t;;
- : char Lwt.state = Return 'a'
\end{lstlisting}
\medskip

\noindent
So, after we write something, the reading thread has been awoken and
has returned the value {\tt 'a'}.



\subsection{ Primitives for thread creation }

There are several primitives for creating {\tt Lwt} threads. These
functions are located in the module {\tt Lwt}.



Here are the main primitives:



\begin{itemize}
\item  {\tt Lwt.return : 'a -> 'a Lwt.t}
\mbox{}\\
creates a thread which has already terminated and returned a value
\item  {\tt Lwt.fail : exn -> 'a Lwt.t}
\mbox{}\\
creates a thread which has already terminated and failed with an
exception
\item  {\tt Lwt.wait : unit -> 'a Lwt.t * 'a Lwt.u}
\mbox{}\\
creates a sleeping thread and returns this thread plus a wakener (of
type {\tt 'a Lwt.u}) which must be used to wakeup the sleeping
thread.

\end{itemize}

To wake up a sleeping thread, you must use one of the following
functions:



\begin{itemize}
\item  {\tt Lwt.wakeup : 'a Lwt.u -> 'a -> unit}
\mbox{}\\
wakes up the thread with a value.
\item  {\tt Lwt.wakeup\_exn : 'a Lwt.u -> exn -> unit}
\mbox{}\\
wakes up the thread with an exception.

\end{itemize}

Note that this is an error to wakeup the same threads twice. {\tt Lwt}
will raise {\tt Invalid\_argument} if you try to do so.



With these informations, try to guess the result of each of the
following expression:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# Lwt.state (Lwt.return 42);;
# Lwt.state (fail Exit);;
# let waiter, wakener = Lwt.wait ();;
# Lwt.state waiter;;
# Lwt.wakeup wakener 42;;
# Lwt.state waiter;;
# let waiter, wakener = Lwt.wait ();;
# Lwt.state waiter;;
# Lwt.wakeup_exn wakener Exit;;
# Lwt.state waiter;;
\end{lstlisting}
\medskip

\noindent
\subsubsection{ Primitives for thread composition }

The most important operation you need to know is {\tt bind}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val bind : 'a Lwt.t -> ('a -> 'b Lwt.t) -> 'b Lwt.t
\end{lstlisting}
\medskip

\noindent
{\tt bind t f} creates a thread which waits for {\tt t} to
terminate, then passes the result to {\tt f}. If {\tt t} is a
sleeping thread, then {\tt bind t f} will be a sleeping thread too,
until {\tt t} terminates. If {\tt t} fails, then the resulting
thread will fail with the same exception. For example, consider the
following expression:



\lstset{language=[Objective]Caml}\begin{lstlisting}
Lwt.bind
  (Lwt_io.read_line Lwt_io.stdin)
  (fun str -> Lwt_io.printlf "You typed %S" str)
\end{lstlisting}
\medskip

\noindent
This code will first wait for the user to enter a line of text, then
print a message on the standard output.



Similarly to {\tt bind}, there is a function to handle the case
when {\tt t} fails:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val catch : (unit -> 'a Lwt.t) -> (exn -> 'a Lwt.t) -> 'a Lwt.t
\end{lstlisting}
\medskip

\noindent
{\tt catch f g} will call {\tt f ()}, then waits for its
termination, and if it fails with an exception {\tt exn}, calls
{\tt g exn} to handle it. Note that both exceptions raised with
{\tt Pervasives.raise} and {\tt Lwt.fail} are caught by
{\tt catch}.



\subsubsection{ Cancelable threads }

In some case, we may want to cancel a thread. For example, because it
has not terminated after a timeout. This can be done with cancelable
threads. To create a cancelable thread, you must use the
{\tt Lwt.task} function:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val task : unit -> 'a Lwt.t * 'a Lwt.u
\end{lstlisting}
\medskip

\noindent
It has the same semantics as {\tt Lwt.wait} except that the
sleeping thread can be canceled with {\tt Lwt.cancel}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val cancel : 'a Lwt.t -> unit
\end{lstlisting}
\medskip

\noindent
The thread will then fail with the exception
{\tt Lwt.Canceled}. To execute a function when the thread is
canceled, you must use {\tt Lwt.on\_cancel}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val on_cancel : 'a Lwt.t -> (unit -> unit) -> unit
\end{lstlisting}
\medskip

\noindent
Note that it is also possible to cancel a thread which has not been
created with {\tt Lwt.task}. In this case, the deepest cancelable
thread connected with the given thread will be cancelled.



For example, consider the following code:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# let waiter, wakener = Lwt.task ();;
val waiter : '_a Lwt.t = <abstr>
val wakener : '_a Lwt.u = <abstr>
# let t = bind waiter (fun x -> return (x + 1));;
val t : int Lwt.t = <abstr>
\end{lstlisting}
\medskip

\noindent
Here, cancelling {\tt t} will in fact cancel {\tt waiter}.
{\tt t} will then fail with the exception {\tt Lwt.Canceled}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# Lwt.cancel t;;
- : unit = ()
# Lwt.state waiter;;
- : int Lwt.state = Fail Lwt.Canceled
# Lwt.state t;;
- : int Lwt.state = Fail Lwt.Canceled
\end{lstlisting}
\medskip

\noindent
By the way, it is possible to prevent a thread from being canceled
by using the function {\tt Lwt.protected}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val protected : 'a Lwt.t -> 'a Lwt.t
\end{lstlisting}
\medskip

\noindent
Canceling {\tt (proctected t)} will have no effect on {\tt t}.



\subsubsection{ Primitives for multi-thread composition }

We now show how to compose several concurrent threads. The
main functions for this are in the {\tt Lwt} module: {\tt join},
{\tt choose} and {\tt pick}.



The first one, {\tt join} takes a list of threads and waits for all
of them to terminate:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val join : unit Lwt.t list -> unit Lwt.t
\end{lstlisting}
\medskip

\noindent
Moreover, if at least one thread fails, {\tt join l} will fail with
the same exception as the first to fail, after all threads terminate.



Similarly {\tt choose} waits for at least one thread to
terminate, then returns the same value or exception:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val choose : 'a Lwt.t list -> 'a Lwt.t
\end{lstlisting}
\medskip

\noindent
For example:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# let waiter1, wakener1 = Lwt.wait ();;
val waiter1 : '_a Lwt.t = <abstr>
val wakener1 : '_a Lwt.u = <abstr>
# let waiter2, wakener2 = Lwt.wait ();;
val waiter2 : '_a Lwt.t = <abstr>
val wakener : '_a Lwt.u = <abstr>
# let t = Lwt.choose [waiter1; waiter2];;
val t : '_a Lwt.t = <abstr>
# Lwt.state t;;
- : '_a Lwt.state = Sleep
# Lwt.wakeup wakener2 42;;
- : unit = ()
# Lwt.state t;;
- : int Lwt.state = Return 42
\end{lstlisting}
\medskip

\noindent
The last one, {\tt pick}, is the same as {\tt join} except that it cancels
all other threads when one terminates.



\subsubsection{ Threads local storage }

Lwt can store variables with different values on different
threads. This is called threads local storage. For example, this can
be used to store contexts or thread identifiers. The contents of a
variable can be read with:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val Lwt.get : 'a Lwt.key -> 'a option
\end{lstlisting}
\medskip

\noindent
which takes a key to identify the variable we want to read and
returns either {\tt None} if the variable is not set, or
{\tt Some x} if it is. The value returned is the value of the
variable in the current thread.



New keys can be created with:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val Lwt.new_key : unit -> 'a Lwt.key
\end{lstlisting}
\medskip

\noindent
To set a variable, you must use:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val Lwt.with_value : 'a Lwt.key -> 'a option -> (unit -> 'b) -> 'b
\end{lstlisting}
\medskip

\noindent
{\tt with\_value key value f} will execute {\tt f} with
the binding {\tt key -> value}. The old value associated to
{\tt key} is restored after {\tt f} terminates.



For example, you can use local storage to store thread identifiers
and use them in logs:



\lstset{language=[Objective]Caml}\begin{lstlisting}
let id_key = Lwt.new_key ()

let log msg =
  let thread_id =
    match Lwt.get id_key with
      | Some id -> id
      | None -> "main"
  in
  Lwt_io.printlf "%s: %s" thread_id msg

lwt () =
  Lwt.join [
    Lwt.with_value id_key (Some "thread 1") (fun () -> log "foo");
    Lwt.with_value id_key (Some "thread 2") (fun () -> log "bar");
  ]
\end{lstlisting}
\medskip

\noindent
\subsubsection{ Rules }

{\tt Lwt} will always try to execute as much as possible before yielding and
switching to another cooperative thread. In order to make it work well,
you must follow the following rules:



\begin{itemize}
\item  do not write function that may takes time to complete without
using {\tt Lwt},
\item  do not do IOs that may block, otherwise the whole program will
hang. You must instead use asynchronous IOs operations.

\end{itemize}

\subsection{ The syntax extension }

{\tt Lwt} offers a syntax extension which increases code readability and
makes coding using {\tt Lwt} easier.  To use it add the ``lwt.syntax'' package when
compiling:



\lstset{language=[Objective]Caml}\begin{lstlisting}
$ ocamlfind ocamlc -syntax camlp4o -package lwt.syntax -linkpkg -o foo foo.ml
\end{lstlisting}
\medskip

\noindent
Or in the toplevel (after loading topfind):



\lstset{language=[Objective]Caml}\begin{lstlisting}
# #camlp4o;;
# #require "lwt.syntax";;
\end{lstlisting}
\medskip

\noindent
The following constructions are added to the language:



\begin{itemize}
\item  {\tt lwt} \emph{pattern$_{\mbox{1}}$} {\tt =} \emph{expr$_{\mbox{1}}$} [ {\tt and}
\emph{pattern$_{\mbox{2}}$} {\tt =} \emph{expr$_{\mbox{2}}$} ... ] {\tt in} \emph{expr}
\mbox{}\\
which is a parallel let-binding construction. For example in the
following code:

\end{itemize}

\lstset{language=[Objective]Caml}\begin{lstlisting}
lwt x = f () and y = g () in
expr
\end{lstlisting}
\medskip

\noindent
the thread {\tt f ()} and {\tt g ()} are launched concurrently
and their results are then bound to {\tt x} and {\tt y} in the
expression \emph{expr}.



Of course you can also launch the two threads sequentially by
writing your code like that:



\lstset{language=[Objective]Caml}\begin{lstlisting}
lwt x = f () in
lwt y = g () in
expr
\end{lstlisting}
\medskip

\noindent
\begin{itemize}
\item  {\tt try\_lwt} \emph{expr} [ {\tt with} \emph{pattern$_{\mbox{1}}$}
{\tt ->} \emph{expr$_{\mbox{1}}$} ... ] [ {\tt finally} \emph{expr'} ]
\mbox{}\\
which is the equivalent of the standard {\tt try-with}
construction but for {\tt Lwt}. Both exceptions raised by
{\tt Pervasives.raise} and {\tt Lwt.fail} are caught.";

\end{itemize}

\begin{itemize}
\item  {\tt for\_lwt} \emph{ident} {\tt =} \emph{expr$_{\mbox{init}}$} ( {\tt to} {\tt |}
{\tt downto} ) \emph{expr$_{\mbox{final}}$} {\tt do} \emph{expr}
{\tt done}
\mbox{}\\
which is the equivalent of the standard {\tt for} construction
but for {\tt Lwt}.

\end{itemize}

\begin{itemize}
\item  {\tt raise\_lwt} \emph{exn}
\mbox{}\\
which is the same as {\tt Lwt.fail} \emph{exn} but with backtrace support.

\end{itemize}

\subsubsection{ Correspondence table }

You might appreciate the following table to write code using lwt:



\noindent
\begin{tabular}{p{0.5\textwidth}p{0.5\textwidth}}
\multicolumn{1}{l}{\begin{minipage}{0.5\textwidth}\centering  without {\tt Lwt}                                                               \end{minipage}}&\multicolumn{1}{l}{\begin{minipage}{0.5\textwidth}\centering  with {\tt Lwt}                                                                      \end{minipage}}\\
                                                                                  &                                                                                      \\
 {\tt let} \emph{pattern$_{\mbox{1}}$} {\tt =} \emph{expr$_{\mbox{1}}$}                                 & {\tt lwt} \emph{pattern$_{\mbox{1}}$} {\tt =} \emph{expr$_{\mbox{1}}$}                                     \\
 {\tt and} \emph{pattern$_{\mbox{2}}$} {\tt =} \emph{expr$_{\mbox{2}}$}                                 & {\tt and} \emph{pattern$_{\mbox{2}}$} {\tt =} \emph{expr$_{\mbox{2}}$}                                     \\
 ...                                                                              & ...                                                                                  \\
 {\tt and} \emph{pattern$_{\mbox{n}}$} {\tt =} \emph{expr$_{\mbox{n}}$} {\tt in}                        & {\tt and} \emph{pattern$_{\mbox{n}}$} {\tt =} \emph{expr$_{\mbox{n}}$} {\tt in}                            \\
 \emph{expr}                                                                         & \emph{expr}                                                                             \\
                                                                                  &                                                                                      \\
 {\tt try}                                                                        & {\tt try\_lwt}                                                                        \\
 \emph{ expr}                                                                        & \emph{ expr}                                                                            \\
 {\tt with}                                                                       & {\tt with}                                                                           \\
 \emph{ } {\tt |} \emph{pattern$_{\mbox{1}}$} {\tt ->} \emph{expr$_{\mbox{1}}$}                            & \emph{ } {\tt |} \emph{pattern$_{\mbox{1}}$} {\tt ->} \emph{expr$_{\mbox{1}}$}                                \\
 \emph{ } {\tt |} \emph{pattern$_{\mbox{2}}$} {\tt ->} \emph{expr$_{\mbox{2}}$}                            & \emph{ } {\tt |} \emph{pattern$_{\mbox{2}}$} {\tt ->} \emph{expr$_{\mbox{2}}$}                                \\
 \emph{ } ...                                                                        & \emph{ } ...                                                                            \\
 \emph{ } {\tt |} \emph{pattern$_{\mbox{n}}$} {\tt ->} \emph{expr$_{\mbox{n}}$}                            & \emph{ } {\tt |} \emph{pattern$_{\mbox{n}}$} {\tt ->} \emph{expr$_{\mbox{n}}$}                                \\
                                                                                  &                                                                                      \\
 {\tt for} \emph{ident} {\tt =} \emph{expr$_{\mbox{init}}$} {\tt to} \emph{expr$_{\mbox{final}}$} {\tt do} & {\tt for\_lwt} \emph{ident} {\tt =} \emph{expr$_{\mbox{init}}$} {\tt to} \emph{expr$_{\mbox{final}}$} {\tt do} \\
 \emph{ expr}                                                                        & \emph{ expr}                                                                            \\
 {\tt done}                                                                       & {\tt done}                                                                           \\
                                                                                  &                                                                                      \\
 {\tt raise} \emph{exn}                                                              & {\tt raise\_lwt} \emph{exn}                                                              \\
                                                                                  &                                                                                      \\
 {\tt assert} \emph{expr}                                                            & {\tt assert\_lwt} \emph{expr}                                                            \\
                                                                                  &                                                                                      \\
 {\tt match} \emph{expr} {\tt with}                                                  & {\tt match\_lwt} \emph{expr} {\tt with}                                                  \\
 \emph{ } {\tt |} \emph{pattern$_{\mbox{1}}$} {\tt ->} \emph{expr$_{\mbox{1}}$}                            & \emph{ } {\tt |} \emph{pattern$_{\mbox{1}}$} {\tt ->} \emph{expr$_{\mbox{1}}$}                                \\
 \emph{ } {\tt |} \emph{pattern$_{\mbox{2}}$} {\tt ->} \emph{expr$_{\mbox{2}}$}                            & \emph{ } {\tt |} \emph{pattern$_{\mbox{2}}$} {\tt ->} \emph{expr$_{\mbox{2}}$}                                \\
 \emph{ } ...                                                                        & \emph{ } ...                                                                            \\
 \emph{ } {\tt |} \emph{pattern$_{\mbox{n}}$} {\tt ->} \emph{expr$_{\mbox{n}}$}                            & \emph{ } {\tt |} \emph{pattern$_{\mbox{n}}$} {\tt ->} \emph{expr$_{\mbox{n}}$}                                \\
                                                                                  &                                                                                      \\
 {\tt while} \emph{expr} {\tt do}                                                    & {\tt while\_lwt} \emph{expr} {\tt do}                                                    \\
 \emph{ expr}                                                                        & \emph{ expr}                                                                            \\
 {\tt done}                                                                       & {\tt done}                                                                           \\
\end{tabular}

\subsection{ Backtrace support }

When using {\tt Lwt}, exceptions are not recorded by the ocaml runtime, and so you don't
get backtraces. However it is possible to get them when using the syntax extension. All you
have to do is to pass the {\tt -lwt-debug} switch to {\tt camlp4}:



\begin{verbatim}
$ ocamlfind ocamlc -syntax camlp4o -package lwt.syntax \
    -ppopt -lwt-debug -linkpkg -o foo foo.ml
\end{verbatim}
\medskip

\noindent
\subsection{ Other modules of the core library }

The core library contains several modules that only depend on
{\tt Lwt}. The following naming convention is used in {\tt Lwt}: when a
function takes as argument a function returning a thread that is going
to be executed sequentially, it is suffixed with ``{\tt \_s}''. And
when it is going to be executed concurrently, it is suffixed with
``{\tt \_p}''. For example, in the {\tt Lwt\_list} module we have:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val map_s : ('a -> 'b Lwt.t) -> 'a list -> 'b list Lwt.t
val map_p : ('a -> 'b Lwt.t) -> 'a list -> 'b list Lwt.t
\end{lstlisting}
\medskip

\noindent
\subsubsection{ Mutexes }

{\tt Lwt\_mutex} provides mutexes for {\tt Lwt}. Its use is almost the
same as the {\tt Mutex} module of the thread library shipped with
OCaml. In general, programs using {\tt Lwt} do not need a lot of
mutexes. They are only usefull for serialising operations.



\subsubsection{ Lists }

The {\tt Lwt\_list} module defines iteration and scanning functions
over lists, similar to the ones of the {\tt List} module, but using
functions that return a thread. For example:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val iter_s : ('a -> unit Lwt.t) -> 'a list -> unit Lwt.t
val iter_p : ('a -> unit Lwt.t) -> 'a list -> unit Lwt.t
\end{lstlisting}
\medskip

\noindent
In {\tt iter\_s f l}, {\tt iter\_s} will call f on each elements
of {\tt l}, waiting for completion between each element. On the
contrary, in {\tt iter\_p f l}, {\tt iter\_p} will call f on all
elements of {\tt l}, then wait for all the threads to terminate.



\subsubsection{ Data streams }

{\tt Lwt} streams are used in a lot of places in {\tt Lwt} and its sub
libraries. They offer a high-level interface to manipulate data flows.



A stream is an object which returns elements sequentially and
lazily. Lazily means that the source of the stream is touched only for new
elements when needed. This module contains a lot of stream
transformation, iteration, and scanning functions.



The common way of creating a stream is by using
{\tt Lwt\_stream.from} or by using {\tt Lwt\_stream.create}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val from : (unit -> 'a option Lwt.t) -> 'a Lwt_stream.t
val create : unit -> 'a Lwt_stream.t * ('a option -> unit)
\end{lstlisting}
\medskip

\noindent
As for streams of the standard library, {\tt from} takes as
argument a function which is used to create new elements.



{\tt create} returns a function used to push new elements
into the stream and the stream which will receive them.



For example:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# let stream, push = Lwt_stream.create ();;
val stream : '_a Lwt_stream.t = <abstr>
val push : '_a option -> unit = <fun>
# push (Some 1);;
- : unit = ()
# push (Some 2);;
- : unit = ()
# push (Some 3);;
- : unit = ()
# Lwt.state (Lwt_stream.next stream);;
- : int Lwt.state = Return 1
# Lwt.state (Lwt_stream.next stream);;
- : int Lwt.state = Return 2
# Lwt.state (Lwt_stream.next stream);;
- : int Lwt.state = Return 3
# Lwt.state (Lwt_stream.next stream);;
- : int Lwt.state = Sleep
\end{lstlisting}
\medskip

\noindent
Note that streams are consumable. Once you take an element from a
stream, it is removed from it. So, if you want to iterate two times
over a stream, you may consider ``clonning'' it, with
{\tt Lwt\_stream.clone}. Cloned stream will return the same
elements in the same order. Consuming one will not consume the other.
For example:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# let s = Lwt_stream.of_list [1; 2];;
val s : int Lwt_stream.t = <abstr>
# let s' = Lwt_stream.clone s;;
val s' : int Lwt_stream.t = <abstr>
# Lwt.state (Lwt_stream.next s);;
- : int Lwt.state = Return 1
# Lwt.state (Lwt_stream.next s);;
- : int Lwt.state = Return 2
# Lwt.state (Lwt_stream.next s');;
- : int Lwt.state = Return 1
# Lwt.state (Lwt_stream.next s');;
- : int Lwt.state = Return 2
\end{lstlisting}
\medskip

\noindent
\subsubsection{ Mailbox variables }

The {\tt Lwt\_mvar} module provides mailbox variables. A mailbox
variable, also called a ``mvar'', is a cell which may contain 0 or 1
element. If it contains no elements, we say that the mvar is empty,
if it contains one, we say that it is full. Adding an element to a
full mvar will block until one is taken. Taking an element from an
empty mvar will block until one is added.



Mailbox variables are commonly used to pass messages between threads.



Note that a mailbox variable can be seen as a pushable stream with a
limited memory.



\section{ Running a Lwt program }

Threads you create with {\tt Lwt} always have the type
{\tt Lwt.t}. If you want to write a program and run it this is not
enough. Indeed you don't know when a {\tt Lwt} thread is terminated.



For example if your program is just:



\lstset{language=[Objective]Caml}\begin{lstlisting}
let _ = Lwt_io.printl "Hello, world!"
\end{lstlisting}
\medskip

\noindent
you have no guarantee that the thread writing {\tt "Hello, world!"}
on the terminal will be terminated when the program exit. In order
to wait for a thread to terminate, you have to call the function
{\tt Lwt\_main.run}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val Lwt_main.run : 'a Lwt.t -> 'a
\end{lstlisting}
\medskip

\noindent
This functions wait for the given thread to terminate and returns
its result. In fact it does more than that; it also run the
scheduler which is responsible for making thread to progress when
events are received from the outside world.



So basically, when you write a {\tt Lwt} program you must call at
the toplevel the function {\tt Lwt\_main.run}. For instance:



\lstset{language=[Objective]Caml}\begin{lstlisting}
let () = Lwt_main.run (Lwt_io.printl "Hello, world!")
\end{lstlisting}
\medskip

\noindent
Note that you must call {\tt Lwt\_main.run} only once at a time. It
cannot be used anywhere to get the result of a thread. It must only
be used in the entry point of your program.



\section{ The {\tt lwt.unix} library }

The package {\tt lwt.unix} contains all {\tt unix} dependent
modules of {\tt Lwt}. Among all its features, it implements cooperative
versions of functions of the standard library and the unix library.



\subsection{ Unix primitives }

The {\tt Lwt\_unix} provides cooperative system calls. For example,
the {\tt Lwt} counterpart of {\tt Unix.read} is:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val read : file_descr -> string -> int -> int -> int Lwt.t
\end{lstlisting}
\medskip

\noindent
{\tt Lwt\_io} provides features similar to buffered channels of
the standard library (of type {\tt in\_channel} or
{\tt out\_channel}) but cooperatively.



{\tt Lwt\_gc} allows you to register a finaliser that returns a
thread. At the end of the program, {\tt Lwt} will wait for all the
finaliser to terminate.



\subsection{ The Lwt scheduler }

Threads doing IO may be put asleep until some events are received by
the process. For example when you read from a file descriptor, you
may have to wait for the file descriptor to become readable if no
data are immediatly available on it.



{\tt Lwt} contains a shceduler which is responsible for managing
multiple threads waiting for events, and restart them when needed.
This scheduler is implemented by the two modules {\tt Lwt\_engine}
and {\tt Lwt\_main}. {\tt Lwt\_engine} is a low-level module, it
provides signatures for IO multiplexers as well as several builtin
implementation. {\tt Lwt} support by default multiplexing IO with
{\tt libev} or {\tt Unix.select}. The signature is given by the
class {\tt Lwt\_engine.t}.



{\tt libev} is used by default on Unix, because it supports any
number of file descriptors while Unix.select supports only 1024 at
most, and is also much more efficient. On Windows {\tt Unix.select}
is used because {\tt libev} does not works properly. The user may
change at any time the backend in use.



The engine can also be used directly in order to integrate other
libraries with {\tt Lwt}. For example {\tt GTK} need to be notified
when some events are received. If you use {\tt Lwt} with {\tt GTK}
you need to use the {\tt Lwt} scheduler to monitor {\tt GTK}
sources. This is what is done by the {\tt lwt.glib} package.



The {\tt Lwt\_main} module contains the \emph{main loop} of
{\tt Lwt}. It is run by calling the function {\tt Lwt\_main.run}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val Lwt_main.run : 'a Lwt.t -> 'a
\end{lstlisting}
\medskip

\noindent
This function continously run the scheduler until the thread passed
as argument terminates.



\subsection{ The logging facility }

The package {\tt lwt.unix} contains a module {\tt Lwt\_log}
providing loggers. It supports logging to a file, a channel, or to the
syslog daemon. You can also define your own logger by providing the
appropriate functions (function {\tt Lwt\_log.make}).



Several loggers can be merged into one. Sending logs on the merged
logger will send these logs to all its components.



For example to redirect all logs to {\tt stderr} and to the syslog
daemon:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# Lwt_log.default_logger :=
    Lwt_log.broadcast [
      Lwt_log.channel ~close_mode:`Keep ~channel:Lwt_io.stderr ();
      Lwt_log.syslog ~facility:`User ();
    ]
;;
\end{lstlisting}
\medskip

\noindent
{\tt Lwt} also provides a syntax extension, in the package
{\tt lwt.syntax.log}. It does not modify the language but
it replaces log statement of the form:



\lstset{language=[Objective]Caml}\begin{lstlisting}
Lwt_log.info_f ~section "something happened: %s" msg
\end{lstlisting}
\medskip

\noindent
by:



\lstset{language=[Objective]Caml}\begin{lstlisting}
if Lwt_log.Section.level section <= Lwt_log.Info then
  Lwt_log.info_f ~section "somethign happend: %s" msg
else
  Lwt.return ()
\end{lstlisting}
\medskip

\noindent
The advantages of using the syntax extension are the following:



\begin{itemize}
\item  it checks the log level before calling the logging function, so
the arguments are not computed if not needed
\item  debugging logs can be removed at parsing time

\end{itemize}

By default, the syntax extension removes all logs with the level
{\tt debug}. To keep them, pass the command line option
{\tt -lwt-debug} to camlp4.



\section{ The Lwt.react library }

The {\tt Lwt\_react} module provides helpers for using the {\tt react}
library with {\tt Lwt}. It extends the {\tt React} module by adding
{\tt Lwt} specific functions. It can be used as a replacement of
{\tt React}. For example you can add at the beginning of you
program:



\lstset{language=[Objective]Caml}\begin{lstlisting}
open Lwt_react
\end{lstlisting}
\medskip

\noindent
instead of:



\lstset{language=[Objective]Caml}\begin{lstlisting}
open React
\end{lstlisting}
\medskip

\noindent
or:



\lstset{language=[Objective]Caml}\begin{lstlisting}
module React = Lwt_react
\end{lstlisting}
\medskip

\noindent
Among the added functionalities we have {\tt Lwt\_react.E.next}, which
takes an event and returns a thread which will wait until the next
occurence of this event. For example:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# open Lwt_react;;
# let event, push = E.create ();;
val event : '_a React.event = <abstr>
val push : '_a -> unit = <fun>
# let t = E.next event;;
val t : '_a Lwt.t = <abstr>
# Lwt.state t;;
- : '_a Lwt.state = Sleep
# push 42;;
- : unit = ()
# Lwt.state t;;
- : int Lwt.state = Return 42
\end{lstlisting}
\medskip

\noindent
Another interesting feature is the ability to limit events
(resp. signals) from occuring (resp. changing) too often. For example,
suppose you are doing a program which displays something on the screeen
each time a signal changes. If at some point the signal changes 1000
times per second, you probably want not to render it 1000 times per
second. For that you use {\tt Lwt\_react.S.limit}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val limit : (unit -> unit Lwt.t) -> 'a React.signal -> 'a React.signal
\end{lstlisting}
\medskip

\noindent
{\tt Lwt\_react.S.limit f signal} returns a signal which varies as
{\tt signal} except that two consecutive updates are separeted by a
call to {\tt f}. For example if {\tt f} returns a thread which sleep
for 0.1 seconds, then there will be no more than 10 changes per
second. For example:



\lstset{language=[Objective]Caml}\begin{lstlisting}
open Lwt_react

let draw x =
  (* Draw the screen *)
  ...

let () =
  (* The signal we are interested in: *)
  let signal = ... in

  (* The limited signal: *)
  let signal' = S.limit (fun () -> Lwt_unix.sleep 0.1) signal in

  (* Redraw the screen each time the limited signal change: *)
  S.notify_p draw signal'
\end{lstlisting}
\medskip

\noindent
\section{ The lwt.text library (deprecated) }

  {\bfseries WARNING:} the {\tt lwt.text} library is deprecated. It has been
replaced by the {\tt lambda-term} library which is more complete and
more portable. It is available here:
\hyperref[http://lambda-term.forge.ocamlcore.org/]{http://lambda-term.forge.ocamlcore.org/}.



The {\tt lwt.text} library provides functions to deal with text
mode (in a terminal). It is composed of the three following modules:



\begin{itemize}
\item  {\tt Lwt\_text}, which is the equivalent of {\tt Lwt\_io}
but for unicode text channels
\item  {\tt Lwt\_term}, providing various terminal utilities, such as
reading a key from the terminal
\item  {\tt Lwt\_read\_line}, which provides functions to input text
from the user with line editing support

\end{itemize}

\subsection{ Text channels }

A text channel is basically a byte channel with an encoding. Input
(resp. output) text channels decode (resp. encode) unicode characters
on the fly. By default, output text channels use transliteration, so
they will not fail because text you want to print cannot be encoded
in the system encoding.



For example, with you locale sets to ``C'', and the variable
{\tt name} set to ``Jérémie'', you got:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# lwt () = Lwt_text.printlf "My name is %s" name;;
My name is J?r?mie
\end{lstlisting}
\medskip

\noindent
\subsection{ Terminal utilities }

The {\tt Lwt\_term} allow you to put the terminal in \emph{raw mode},
meaning that input is not buffered and character are
returned as the user types them. For example, you can read a key with:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# lwt key = Lwt_term.read_key ();;
val key : Lwt_term.key = Lwt_term.Key_control 'j'
\end{lstlisting}
\medskip

\noindent
The second main feature of {\tt Lwt\_term} is the ability to print
text with styles. For example, to print text in bold and blue:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# open Lwt_term;;
# lwt () = printlc [fg blue; bold; text "foo"];;
foo
\end{lstlisting}
\medskip

\noindent
If the output is not a terminal, then {\tt printlc} will drop
styles, and act as {\tt Lwt\_text.printl}.



\subsection{ Read-line }

{\tt Lwt\_read\_line} provides a full featured and fully
customisable read-line implementation. You can either use the
high-level and easy to use {\tt read\_*} functions, or use the
advanced {\tt Lwt\_read\_line.Control.read\_*} functions.



For example:



\lstset{language=[Objective]Caml}\begin{lstlisting}
# open Lwt_term;;
# lwt l = Lwt_read_line.read_line ~prompt:[text "foo> "] ();;
foo> Hello, world!
val l : Text.t = "Hello, world!"
\end{lstlisting}
\medskip

\noindent
The second class of functions is a bit more complicated to use, but
allow to control a running read-line instance. For example you can
temporary hide it to draw something, you can send it commands, fake
input, and the prompt is a signal so it can change dynamically.



\section{ Other libraries }

\subsection{ Detaching computation to preemptive threads }

It may happen that you want to run a function which will take time to
compute or that you want to use a blocking function that cannot be
used in a non-blocking way. For these situations, {\tt Lwt} allow you to
\emph{detach} the computation to a preemptive thread.



This is done by the module {\tt Lwt\_preemptive} of the
{\tt lwt.preemptive} package which maintains a pool of system
threads. The main function is:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val detach : ('a -> 'b) -> 'a -> 'b Lwt.t
\end{lstlisting}
\medskip

\noindent
{\tt detach f x} will execute {\tt f x} in another thread and
asynchronously wait for the result.



If you have to run {\tt Lwt} code in another thread, you must use
the function {\tt Lwt\_preemptive.run\_in\_main}:



\lstset{language=[Objective]Caml}\begin{lstlisting}
val run_in_main : (unit -> 'a Lwt.t) -> 'a
\end{lstlisting}
\medskip

\noindent
It works as follow:



\begin{itemize}
\item  it sends the function to the main thread and wait
\item  the main thread execute the function
\item  when it terminates the main thread sends back the result
\item  the result is returned

\end{itemize}

Note that you cannot call {\tt Lwt\_main.run} in another system
thread, so you must use this function.



\subsection{ SSL support }

The package {\tt lwt.ssl} provides the module {\tt Lwt\_ssl}
which allow to use SSL asynchronously



\subsection{ Glib integration }

The {\tt lwt.glib} embeds the {\tt glib} main loop into the
{\tt Lwt} one. This allows you to write GTK application using {\tt Lwt}.  The
one thing you have to do is to call {\tt Lwt\_glib.install} at
the beginning of you program.



\section{ Writing stubs using {\tt Lwt} }

\subsection{ Thread-safe notifications }

If you want to notify the main thread from another thread, you can use the {\tt Lwt}
thread safe notification system. First you need to create a notification identifier
(which is just an integer) from the OCaml side using the
{\tt Lwt\_unix.make\_notification} function, then you can send it from either the
OCaml code with {\tt Lwt\_unix.send\_notification} function, or from the C code using
the function {\tt lwt\_unix\_send\_notification} (defined in {\tt lwt\_unix\_.h}).



Notifications are received and processed asynchronously by the main thread.



\subsection{ Jobs }

For operations that can not be executed asynchronously, {\tt Lwt}
uses a system of jobs that can be executed in a different threads. A
job is composed of three functions:



\begin{itemize}
\item  A stub function to create the job. It musts allocate a new job
structure and fill its [worker] and [result] fields. This
function is executed in the main thread.
The return type for the OCaml external must be of the form {\tt 'a job}.
\item  A function which executes the job. This one may be executed asynchronously
in another thread. This function must not:

\begin{itemize}
\item  access or allocate OCaml block values (tuples, strings, ...),
\item  call OCaml code.

\end{itemize}
\item  A function which reads the result of the job, free resources and
return the result as an OCaml value. This function is executed in
the main thread.

\end{itemize}

With {\tt Lwt < 2.3.3}, 4 functions (including 3 stubs) were
required. It is still possible to use this mode but it is
deprecated.



We show as example the implementation of {\tt Lwt\_unix.mkdir}. On the C
side we have:



\lstset{language=c}\begin{lstlisting}/**/
/* Structure holding informations for calling [mkdir]. */
struct job_mkdir {
  /* Informations used by lwt.
     It must be the first field of the structure. */
  struct lwt_unix_job job;
  /* This field store the result of the call. */
  int result;
  /* This field store the value of [errno] after the call. */
  int errno_copy;
  /* Pointer to a copy of the path parameter. */
  char* path;
  /* Copy of the mode parameter. */
  int mode;
  /* Buffer for storing the path. */
  char data[];
};

/* The function calling [mkdir]. */
static void worker_mkdir(struct job_mkdir* job)
{
  /* Perform the blocking call. */
  job->result = mkdir(job->path, job->mode);
  /* Save the value of errno. */
  job->errno_copy = errno;
}

/* The function building the caml result. */
static value result_mkdir(struct job_mkdir* job)
{
  /* Check for errors. */
  if (job->result < 0) {
    /* Save the value of errno so we can use it
       once the job has been freed. */
    int error = job->errno_copy;
    /* Copy the contents of job->path into a caml string. */
    value string_argument = caml_copy_string(job->path);
    /* Free the job structure. */
    lwt_unix_free_job(&job->job);
    /* Raise the error. */
    unix_error(error, "mkdir", string_argument);
  }
  /* Free the job structure. */
  lwt_unix_free_job(&job->job);
  /* Return the result. */
  return Val_unit;
}

/* The stub creating the job structure. */
CAMLprim value lwt_unix_mkdir_job(value path, value mode)
{
  /* Get the length of the path parameter. */
  mlsize_t len_path = caml_string_length(path) + 1;
  /* Allocate a new job. */
  struct job_mkdir* job =
    (struct job_mkdir*)lwt_unix_new_plus(struct job_mkdir, len_path);
  /* Set the offset of the path parameter inside the job structure. */
  job->path = job->data;
  /* Copy the path parameter inside the job structure. */
  memcpy(job->path, String_val(path), len_path);
  /* Initialize function fields. */
  job->job.worker = (lwt_unix_job_worker)worker_mkdir;
  job->job.result = (lwt_unix_job_result)result_mkdir;
  /* Copy the mode parameter. */
  job->mode = Int_val(mode);
  /* Wrap the structure into a caml value. */
  return lwt_unix_alloc_job(&job->job);
}\end{lstlisting}
\medskip

\noindent
and on the ocaml side:



\lstset{language=[Objective]Caml}\begin{lstlisting}
(* The stub for creating the job. *)
external mkdir_job : string -> int -> unit job = "lwt_unix_mkdir_job"

(* The ocaml function. *)
let mkdir name perms = Lwt_unix.run_job (mkdir_job name perms)
\end{lstlisting}
\medskip

\noindent