sanlock - shared storage lock manager
=====================================

sanlock is a lock manager for shared storage environments. It allows
applications running on multiple hosts to coordinate access to shared
resources, such as data objects on shared storage systems like a SAN,
preventing data corruption and ensuring data integrity.

sanlock uses the shared storage itself to hold lock state.  The lock state
is typically located on the same storage as the application data that the
locks are protecting.  A distributed application wishing to use sanlock
for coordination will reserve space on the shared storage to hold the lock
state, and pass the location for lock storage to sanlock.

Resources
---------

A sanlock resource is an object which can be locked.  It has a name and a
location on disk.  The resource exists as a 1MB area on shared storage,
which holds the lock state for that resource.  An application initializes
a resource, giving it a name that corresponds to an application-specific
entity.  The application can acquire and release a lock on the resource to
coordinate application activity.  Within sanlock, the process of acquiring
or releasing the lock involves reading and writing to the resource area on
the shared storage.  When sanlock acquires a lock on a resource, the local
host ID is written as the lock owner in the first sector of the resource
area.

Lockspaces
----------

A sanlock lockspace is a collection of resources, and a group of hosts
that can acquire locks on them.  The lockspace has a unique name which is
referenced by each resource in the lockspace.  The lockspace exists as a
1MB area on shared storage, typically preceding the resource areas (also
1MB each.)  The lockspace area consists of 2000 special "host_id leases"
(one sector each).  This supports up to 2000 hosts using the lockspace.
The lease for host_id N exists in sector N-1 of the lockspace disk area.

A lockspace also refers to a group of hosts with common access to the
lockspace's shared storage, and using the lockspace's resources.  Each
host using the lockspace has a unique host name and a unique integer host
ID.  A host must "join" the lockspace before it can lock resources.  The
process of joining the lockspace involves acquiring a special "host_id
lease" within the lockspace disk area.  Acquiring the host_id lease
involves writing the host name to the host_id lease.  A host_id lease is
renewed (once every 20 seconds by default).  The host_id lease renewal
applies to all resource locks held in the lockspace, i.e. renewing the
host_id lease renews all resource locks.  During renewal, a host also
reads the state of other host_id leases in the lockspace.  Monitoring
lease renewals enables a host to determine if other hosts are alive.

Hosts
-----

A host is a machine with a unique name and host ID (an integer in the
range 1-2000).  The host ID is assigned by the application or user to each
host.  The host name can be configured by the user in sanlock.conf,
otherwise it is taken from the product_uuid of the system.  If a name is
not assigned, and product_uuid is not available, sanlock generates a new
random uuid for the host name each time sanlock is started.

To use sanlock, a host must run the sanlock(8) and wdmd(8) daemons, and
have a connection to the shared storage holding lockspaces and resources.
Applications use libsanlock to connect to the sanlock daemon, and make API
calls to initialize lockspaces and resources, join and leave lockspaces,
and acquire and release locks on resources.

When a host has joined a lockspace, and the sanlock daemon has begun
renewing the lockspace host_id lease, the reported host state is "LIVE"
for the given lockspace.  The host's state in other lockspaces depends on
whether the application on the host has joined them, and whether sanlock
maintains access to the lockspace storage for renewals.

If a host stops renewing its host_id lease, its resource locks in that
lockspace will expire, and sanlock will eventually allow those locks to be
acquired by other hosts.  When a host fails to renew its host_id lease for
a period of time (id_renewal_fail_seconds, default 80 seconds), it is
initially placed into the "FAIL" state.  In this state, sanlock attempts
to stop the application from using resources in the lockspace, so that the
resource locks are no longer needed.

If the application shutdown is successful, resource locks and the host_id
lease are dropped, and sanlock stops attempting to renew them.  If not
successful, the failed host will be fenced and then enter the "DEAD"
state.  When dead, a host's unreleased locks will be granted by sanlock on
other hosts.  sanlock includes built-in watchdog-based fencing, and also
supports applications deploying other (often faster) fencing methods, e.g.
removing persistent reservations.

Lock algorithms
---------------

Resource locks are implemented using two alternative methods, depending on
the capabilities of the storage device:

1. The SCSI COMPARE AND WRITE (CAW) command.  This special-purpose
   command, supported by certain storage devices, atomically updates
   a disk sector.  sanlock uses this to atomically update the lock
   owner field in a resource.  If multiple hosts attempt to become
   the lock owner simultaneously, the CAW command will succeed on
   only one of the hosts.

2. The Disk Paxos algorithm.  All hosts which want to become the lock
   owner cooperatively determine, using a sequence of reads and writes,
   which host will become the new lock owner.  The resource lock owner
   is updated with a new value in a way that all hosts recognize the
   same result.

Lockspace host_id leases are also implemented using two alternative
methods:

1. The SCSI COMPARE AND WRITE (CAW) command.

2. A time-delay algorithm.

In theory, host_id leases in a lockspace should not be contended, because
the host_id should be configured uniquely for all hosts.  But, the
algorithm for host_id lease ownership attempts to detect conflict in case
of misconfiguration.  The time delay method is particularly inefficient at
this, which can cause long delays joining a lockspace.  The CAW method has
no delay.

Within sanlock, all resource locks are referred to as "paxos leases" (even
when CAW is used), and all lockspace host_id leases are referred to as
"delta leases" (regardless of which algorithm is used.)

When the shared storage supports the COMPARE AND WRITE (CAW) command (also
known as atomic test and set), it is the most quick and efficient option,
and is recommended.  When CAW is not supported, the Disk Paxos algorithm
is used.  The use of CAW is determined when a resource or lockspace is
initialized on disk by testing if the storage location for the lease
supports CAW.

The amount of I/O required to acquire/release a resource lock depends on:

- the algorithm (CAW or Disk Paxos)
- the lock mode (exclusive or shared)
- the lock state (if the lock is already owned)
- lock contention (are multiple hosts attempting to acquire the lock at the same time)

The I/O required for CAW and Disk Paxos (DP), exclusive and shared locks,
assuming the lock is unowned and uncontended:

```
acquire ex CAW resource lock:
  read (1MB), ioctl (512b)
release ex CAW resource lock:
  ioctl (512b)

acquire sh CAW resource lock:
  read (1MB), ioctl (512b); write (512b); ioctl (512b)
release sh CAW resource lock:
  write (512b)

(ioctl: ioctl submits an SG_IO COMPARE AND WRITE command to a single
sector.  Two sectors of data, previous and next, are sent to the device.)

acquire ex DP resource lock:
  read (1MB), write (512b), read (1MB), write (512b), read (1MB), write (512b)
release ex DP resource lock:
  read (512b), write (512b)

acquire sh DP resource lock
  read (1MB), write (512b), read (1MB), write (512b), read (1MB), write (512b);
  write (512b); read (512b), write (512b)
release sh DP resource lock:
  write (512b)
```


Disk sectors and lease areas
----------------------------

sanlock commonly uses atomic reads and writes of disk sectors.  Storage
with either 512 or 4096 byte sectors is supported.  For 512 byte sectors,
resource and lockspace areas are always 1MB (supporting 2000 hosts).  With
4096 byte sectors, resource and lockspace areas default to 8MB (also
supporting 2000 hosts.)  Discussion elsewhere referring to 1MB disk areas
should be translated to 8MB in this case.  The disk area size of a
lockspace or resource is called "align_size" in sanlock interfaces and
internally.

For disks with 4K sectors, applications can choose to initialize resource
and lockspace areas as 1MB, 2MB, 4MB or 8MB.  The reduced sizes of 1/2/4
MB limit the permitted host_id range (and the max number of hosts) to
250/500/1000.

Accepted combinations of sector size and align size, and the corresponding
max_hosts (and max host_id) are:

```
sector_size 512,  align_size 1M, max_hosts 2000
sector_size 4096, align_size 1M, max_hosts 250
sector_size 4096, align_size 2M, max_hosts 500
sector_size 4096, align_size 4M, max_hosts 1000
sector_size 4096, align_size 8M, max_hosts 2000
```

Using sanlock on shared block devices that do host based mirroring or
replication is unlikely to work correctly.

sanlock can also be used on shared files, e.g. over NFS, but this is not the
primary use case.  When using sanlock on shared files, all sanlock io
should go to one file server.


Disk area example
-----------------
```
/dev/sdx
---------------------------------------
Lockspace | Resource | Resource | ... |
---------------------------------------
0          1MB        2MB        3MB

Lockspace
------------------------------------------------------------
| delta lease  | delta lease  | ... | delta lease  | unused |
| LS name FOO  | LS name FOO  | ... | LS name FOO  |        |
| host_id 1    | host_id 2    | ... | host_id 2000 |        |
| generation 1 | generation 1 | ... | generation 0 |        |
| hostname A   | hostname B   | ... | hostname -   |        |
| timestamp 44 | timestamp 25 | ... | timestamp 0  |        |
-------------------------------------------------------------
0              512              ... 1023488                 1MB

Resource
-------------------------------------------------------------
| paxos lease                                               |
| LS name FOO                                               |
| RS name FILE0                                             |
| owner: host_id 1 generation 1                             |
-------------------------------------------------------------
1MB                                                         2MB

Resource
-------------------------------------------------------------
| paxos lease                                               |
| LS name FOO                                               |
| RS name FILE1                                             |
| owner: host_id 2 generation 1                             |
-------------------------------------------------------------
2MB                                                         3MB
```

- Lockspace "FOO" at the start of the shared disk is
  followed by two resources "FILE0" and "FILE1".

- Two hosts: "A" using host_id 1 and "B" using host_id 2
  have joined the lockspace.

- In joining the lockspace, A and B have written their
  hostname to the host_id lease, and have begun updating
  the timestamp field in each renewal (current times on
  A/B were 44/25 at their last renewals.)

- A has locked FILE0, and B has locked FILE1.

The following sanlock commands will produce the state shown in the
diagram.  These steps illustrate, using command line interfaces to
libsanlock APIs, what an application would typically perform using the
libsanlock APIs.

```
Shared disk /dev/sdx is attached to hosts A and B.

host A initializes disk areas for one lockspace and two resources:
$ sanlock init -s FOO:0:/dev/sdx:0
$ sanlock init -r FOO:FILE0:/dev/sdx:1M
$ sanlock init -r FOO:FILE1:/dev/sdx:2M

host A joins lockspace FOO using host_id 1:
$ sanlock add_lockspace -s FOO:1:/dev/sdx:0
host B joins lockspace FOO using host_id 2:
$ sanlock add_lockspace -s FOO:2:/dev/sdx:0

host A acquires lock on FILE0 (lock associated with the sleep process):
$ sanlock command -r FOO:FILE0:/dev/sdx:1M -c /bin/sleep 10
host B acquires lock on FILE1 (lock associated with the sleep process):
$ sanlock command -r FOO:FILE1:/dev/sdx:2M -c /bin/sleep 10

host A fails to acquire lock on FILE1 (owned by host B):
$ sanlock command -r FOO:FILE1:/dev/sdx:1M -c /bin/sleep 10
host B fails to acquire lock on FILE0 (owned by host A):
$ sanlock command -r FOO:FILE0:/dev/sdx:1M -c /bin/sleep 10

When the sleep processes exit, their associated locks (FILE0 and FILE1)
are automatically released.  (The sleep process is a trivial replacement
for what would typically be a long running application process holding
locks.)

host A and B leave the lockspace:
$ sanlock rem_lockspace -s FOO:1:/dev/sdx:0
$ sanlock rem_lockspace -s FOO:2:/dev/sdx:0
```

Failures
--------

_application process failure_

If a process using sanlock exits, sanlock automatically drops the resource
locks that the process was holding.  This response assumes that if the
process is not running, it can no longer write to the shared storage, so
the locks it held are no longer required.  However, if an application may
be writing to shared storage from a process other than the one connected
to sanlock, then PERSISTENT resource locks should be used, which sanlock
will not automatically drop if the connection to the application process
is closed.

_sanlock daemon failure_

If the sanlock daemon process exits abnormally, e.g. it crashes or is
killed, it will no longer be renewing host_id leases in lockspaces that
had been joined.  To other hosts this will appear to be, and will be
handled as, a host failure.  The host with the failed sanlock daemon will
be reset by its watchdog (since watchdog renewals are tied to lockspace
lease renewals), and/or other hosts will apply a form of external fencing,
e.g. removing persistent reservations from the host with the
crashed/killed sanlock daemon.

_wdmd daemon failure_

sanlock uses the wdmd process to manage and update the local /dev/watchdog
device.  If the wdmd process exits abnormally, e.g. it crashes or is
killed, it will no longer be updating the watchdog, which will lead to the
host being reset by its watchdog.  Other hosts wait until the watchdog has
fired, or until external fencing has occurred, before considering the host
dead.

_host system freeze_

If a host system freezes, sanlock and wdmd will no longer be updating the
watchdog, causing the system to be reset by the watchdog.  Other hosts
wait until the watchdog expiration time, or until external fencing has
occurred, before considering the host dead.

_host system failure_

If a host system fails, e.g. loss of power, other hosts cannot distinguish
the failure from a host system freeze, and will wait for the watchdog
expiration time, or for external fencing to occur, before considering the
host dead.

_renewal i/o failure_

Renewing host_id leases in a lockspace (periodically writing a new
timestamp to the lease) is how other hosts determine liveness.  If renewal
I/O fails to the host_id lease on the shared storage, the host will no
longer appear to be alive.  Other hosts cannot distinguish this condition
from a host system failure or freeze.

Once sanlock has failed to renew a host_id lease for specific period of
time (id_renewal_fail_seconds, default 80 seconds), it begins responding:

- Graceful shutdown.  sanlock will execute a "graceful shutdown" program
if the application has specified one to be run.  The shutdown program
tells the application to shut down because its leases are expiring.  The
application must respond by stopping its activities and releasing its
leases (or exit).

- SIGTERM.  If an application does not specify a graceful shutdown
program, sanlock sends SIGTERM to the process instead.  The process must
release its leases or exit in a prescribed amount of time, or sanlock
proceeds to the next method of stopping.  (An application may specify that
SIGTERM is not appropriate for shutdown and should not be tried.)

- SIGKILL.  sanlock sends SIGKILL to the process using the expiring
leases.  The process has a fixed amount of time to exit after receiving
SIGKILL (note: a process could be stuck waiting for I/O in the kernel and
not killable.)  If it does not exit in time, sanlock will proceed to the
next method.  As with SIGTERM, this method may be disabled by applications
for which SIGKILL is not appropriate, e.g. those that modify shared
storage from processes other than the process connected to sanlock.

- Watchdog reset.  Eventually, if an application has not voluntarily
released all of its locks, sanlock intentionally allows the local watchdog
to do a forced reset of the host.  sanlock carefully manages the timing of
the watchdog device so that it fires shortly before any other host could
take over the resource locks held by local processes.

- External fencing.  Other hosts may still use an external fencing method
against a failed host.  External fencing may finish faster than the
watchdog reset, allowing faster takeover of locks/resources from a failed
host.  The watchdog reset can be optionally disabled with external
fencing.


Watchdog
--------

sanlock uses the wdmd(8) daemon to access /dev/watchdog.  wdmd multiplexes
multiple timeouts onto the single watchdog timer.  This is required
because host_id leases for each lockspace are renewed and expire
independently.

sanlock maintains a wdmd connection for each lockspace host_id lease being
renewed.  Each connection has an expiry time for some seconds in the
future.  After each successful host_id lease renewal, the expiry time is
renewed for the associated wdmd connection.  If wdmd finds any connection
expired, it will not renew the /dev/watchdog timer.  Given enough
successive failed renewals, the watchdog device will fire and reset the
host.  (Given the multiplexing nature of wdmd, shorter overlapping renewal
failures from multiple lockspaces could cause spurious watchdog firing.)

The direct link between host_id lease renewals and watchdog renewals
provides a predictable watchdog firing time based on host_id lease renewal
timestamps that are visible from other hosts.  sanlock knows the time the
watchdog on another host has fired based on the host_id lease time.
Furthermore, if the watchdog device on another host fails to fire when it
should, the continuation of host_id lease renewals from the other host will
make this evident and prevent leases from being taken from the failed
host.

If sanlock is able to stop/kill all processing using an expiring
lockspace, the associated wdmd connection for that lockspace is removed.
The expired wdmd connection will no longer block /dev/watchdog renewals,
and the host should avoid being reset.


External fencing
----------------

External fencing allows an application to control when failed hosts are
declared dead.  When sanlock indicates that a host has entered the failed
state (after missing renewals for id_renewal_fail_seconds), the
application can detect this, and respond by performing a custom form of
fencing that it manages itself.

The custom fencing method would protect the shared storage that the
application is using (sanlock itself is not aware of all the shared
storage that the application may be writing to, it is only aware of the
shared storage used by the lockspace and resources.)  One example of
custom fencing is the application removing persistent reservations of the
failed host.

After the application has successfully fenced the failed host, it notifies
sanlock that the host is dead.  sanlock records this in the host_id lease
of the dead host (the DEAD_EXT flag), which in turn allows resource locks
owned by the dead host to be granted by other hosts.

If the application wants to use external fencing only, and does not want
the watchdog expiration to place a host into the DEAD state, it can pass
the NO_TIMEOUT flag when initializing the lockspace.  With this flag
encoded in the lockspace host_id leases, the host_id leases will enter the
FAIL state based on missing renewals, but the host will only enter the
DEAD state based on application's "external dead" API call.


Timing
------

The sanlock timing parameters control host_id lease renewal frequency,
failure detection time, and total recovery time.  These parameters work
together to balance responsiveness to failures (allowing other hosts to
take over resources from failed hosts) against tolerance for temporary
storage slowdowns (avoid resetting hosts that are temporarily slow.)

*io_timeout*

This is the maximum time allowed for a single I/O operation to complete.
This includes I/O used to renew host_id leases, and I/O used to acquire
resource locks.  If I/O times out while acquiring a resource lock, an
error is returned to the application's lock request.  If I/O times out
while renewing a host_id lease, the renewal I/O continues to be retried
until successful or until the lockspace is removed.

The default io_timeout is 10 seconds.  This value can be configured by the
application when initializing a lockspace, or can be configured in
sanlock.conf.  The io_timeout should be set based on the expected I/O
performance of the shared storage, including potential transient delays.

*id_renewal_seconds*

This is the interval between successful host_id lease renewals.  It
represents the maximum time that a single host_id lease renewal can take
(when each I/O operation takes the maximum io_timeout).

id_renewal_seconds = 2 * io_timeout

With the default io_timeout of 10 seconds, id_renewal_seconds is 20
seconds.  This is the typical interval seen between renewal timestamps
written to disk when I/O completes quickly.  Each host_id lease renewal
consists of one read and one write.

If a renewal completes in less than id_renewal_seconds, sanlock will delay
before beginning the next renewal attempt, so that renewals occur at
approximately id_renewal_seconds intervals.  If a renewal takes the full
id_renewal_seconds to complete, the next renewal attempt will begin
immediately.

*id_renewal_fail_seconds*

This is the time from the last successful host_id lease renewal until
sanlock enters recovery mode and begins attempting to stop local processes
using the expiring lockspace.

id_renewal_fail_seconds = 8 * io_timeout

With the default io_timeout of 10 seconds, id_renewal_fail_seconds is 80
seconds.

When sanlock has been unable to renew a host_id lease for
id_renewal_fail_seconds, it begins recovery procedures: running graceful
shutdown programs, sending SIGTERM/SIGKILL, and ultimately allowing the
watchdog to reset the host if the application using the lockspace does not
shut down in time.

*watchdog_fire_timeout*

This is the time the /dev/watchdog device will wait without being renewed
before firing and resetting the host.  The watchdog_fire_timeout is a
constant, typically 60 seconds.

All hosts using sanlock must use the same watchdog_fire_timeout value.
Mismatched values will break the protection provided by the watchdog.

Some watchdog devices support configurable timeouts.  When using a
configurable watchdog, common alternative values are 30 seconds and 10
seconds (typically paired with io_timeout values of 5 and 2 seconds
respectively.)

*host_dead_seconds*

(Note: host_dead_seconds does not apply to host_id leases with NO_TIMEOUT
set.  Hosts using NO_TIMEOUT rely on an external host dead call from the
application before they are considered dead.  An external host dead
indicator may often be faster than waiting for host_dead_seconds.)

This is the time from the last successful host_id lease renewal until
other hosts can be certain that the host has been reset by its watchdog
and is dead (no longer writing to storage).

host_dead_seconds = id_renewal_fail_seconds + watchdog_fire_timeout

With the default values of io_timeout 10 and watchdog_fire_timeout 60,
host_dead_seconds is 140 seconds (80 + 60).

When a host attempts to acquire a resource lock that is held by another
host, it checks the host_id lease of the owner host.  If the host_id lease
has not been renewed for host_dead_seconds, the resource lock is
considered expired and can be acquired.  Before this time, the resource
lock remains owned by the other host, even if that host has stopped
renewing its host_id lease.

Mailing list
------------

https://lists.fedorahosted.org/admin/lists/sanlock-devel.lists.fedorahosted.org/

