## Modes of operation
casd can be launched in one of two modes:

* As a local **CAS server**, or

* as a **CAS proxy/cache** pointing to one or more remote CAS server/s.

In addition to the Remote Execution API CAS service calls, casd also provides the [LocalCAS protocol](https://gitlab.com/BuildGrid/buildbox/buildbox/blob/master/protos/build/buildgrid/local_cas.proto) methods.

### Server mode
In server mode casd works like a regular CAS implementation.

### Proxy-cache mode
When started in proxy mode, casd is pointed to a remote CAS server and will seek to minimize the data that needs to be transferred to and from that remote by caching data locally.

To start casd in proxy mode, a remote endpoint needs to be specified via the `--cas-remote=URL` and `--cas-instance=NAME` command line options.

In addition, casd supports opening connections to more servers dynamically. For that clients can invoke the LocalCAS protocol method `GetInstanceNameForRemoteRequest()` with the remote's connection details. That will return a string that clients must set in the `instance_name` field of all further requests that they want to be forwarded to that specific remote destination.

If something is stored in the local CAS, but not in the proxy, by default casd will upload to remote to ensure consistency. This can be disabled via the command line flag `--read-only-remote` in the case of a read-only proxy.

### ActionCache functionality
Casd also starts a concurrent ActionCache instance which by default caches locally.
The cache can also be configured to proxy in the same way as the base CAS server.
The command line interface is similar, with the `ac-` prefix instead of `cas-`
e.g. a remote endpoint is specified via `--ac-remote=URL`.  

If something is stored in the local ActionCache, but not in the proxy, by default casd will upload to remote to ensure consistency. This can be disabled via the command line flag `--read-only-remote` in the case of a read-only proxy.

#### Auxiliary server instance
When launching casd in proxy mode it is also possible to create an additional server instance by using the `--local-server-instance` option to name it. The server instance will run alongside the proxy and, since they share a common storage, both instances will have access to the same blobs.

This allows configuring casd to have an instance with a name such as `"proxy"` that is connected to a remote and a `"server"` instance that does not perform any forwarding. Clients can then switch specific requests to the `"server"` instance in order to store private blobs that need not be propagated to a remote CAS.

An application of that configuration could be when a remote worker pre-computes a directory structure such as a base image and wants casd to stage it. In that case the contents of that directory must be added to CAS, but having those blobs propagated to the remote CAS might be inefficient.


#### Call behaviors
##### `FindMissingBlobs()`
The request is forwarded as-is to the remote CAS.

For each digest reported missing by the remote, checks whether a copy of that blob is stored locally and in that case attempts to upload it to the remote. If the update is successful that digest is not included in the return list.

The returned list is guaranteed to contain the subset of digests that are still missing in the remote.

###### Result caching
If the `--findmissingblobs-cache-ttl` option is set to a positive value, the list of digests that the remote reports as present will be cached for that number of seconds.

That assumes that if a digest is present in the remote it will remain so for at least that period of time, and can help reduce the number of round-trips to/from the remote CAS.

In addition, when a blob is successfully uploaded to a remote, its digest is also added to this cache.

##### `BatchReadBlobs()`
Only fetches from the remote digests that are not present locally and adds them to the storage (so subsequent calls will return the local copy).

##### `BatchUpdateBlobs()`
Forwards the request as-is to the remote and saves a local copy of each blob.

**Note: this can be optimized by issuing a `FindMissingBlobs()` call before uploading.**

##### `GetTree()`
This call follows the logic for the ByteStream `Read()` and `Write()` for each blob in the tree.

##### Bytestream
###### `Read()`
If the data is present locally it is served from the local copy. Otherwise forwards the request to the remote and, if successful, adds the result to the local storage.

###### `Write()`
Forwards the request to the remote. If the upload succeeds and the data is not already present, adds a copy of the blob to the local storage.

### LocalCAS protocol
casd implements the [LocalCAS protocol](https://gitlab.com/BuildGrid/buildbox/buildbox/blob/master/protos/build/buildgrid/local_cas.proto) methods.

#### `StageTree()`
This method makes the specified directory tree temporarily available for local filesystem access.

Currently, casd supports two mechanisms to implement the operation.

##### Hard links
Files are staged by creating read-only hard links to the actual blobs stored in `LOCAL_CACHE_PATH`. This is simple and avoids doing copies. However, a limitation of relying on hard links is that it only supports staging into directories inside the same filesystem that hosts the `LOCAL_CACHE_PATH` directory.

###### Permissions
By default casd will set the root of the staged directory to mode 0777.

This may not be suitable for some use cases. For example, [buildbox-run-userchroot](https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/run-userchroot/) requires a root directory to be staged with mode 0755 to meet the requirements imposed by `userchroot`. In order to support that scenario, casd will check the root directory for a `unix_mode` property set to `0755`. Note that it is the only value allowed and it is only considered for the root directory.

##### FUSE
casd can also employ [buildbox-fuse](https://gitlab.com/BuildGrid/buildbox/-/blob/master/fuse/) to stage directories using *Filesystem in Userspace* (FUSE). This provides more flexibility, allowing to stage directories in a different filesystem.

**Note:** casd will automatically use this mechanism if the `buildbox-fuse` binary is located when searching using the `$PATH` environment variable.

#### `CaptureTree()`
This operation reads a whole directory and adds it into the CAS.

The response will contain the `Digest` of a `Tree` message representing that directory, which can later be used to fetch or stage the directory.

# Storage
All blobs are stored on disk in the specified `LOCAL_CACHE_PATH` directory that will have this structure:
* `CAS_PATH/cas/objects`: contains subdirectories named after the first two most-significant hex digits of the digest. The blobs are stored in those directories in files named after the rest of the digits.

* `CAS_PATH/cas/tmp`: that directory is used to create temporary files used during writes (see the *Concurrency* section)

For example, the blob for digest `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` will be in `CAS_PATH/cas/objects/e3/b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`

**Note:** External processes are allowed to access blobs directly from the `LOCAL_CACHE_PATH` directory. However, that is limited to **read-only** operations.

#### `FetchTree()`
If a remote CAS is configured (proxy mode), this method fetches an entire directory tree from a remote CAS (equivalent to calling `GetTree()` and storing the `Directory` objects in the local cache). It can also fetch all blobs referenced by the `Directory` objects (equivalent to `FetchMissingBlobs()`).

If no remote CAS is available (server mode), it will check presence of the entire directory tree, and optionally its file blobs, in the local cache.

In order to avoid downloading the same tree multiple times in parallel, casd will consolidate `FetchTree()` requests for a same root digest that arrive at the same time.

To further improve the performance of this call, casd will cache the status that indicates that a tree is present in Local CAS, which reduces the number of storage scans needed to check whether a whole tree is present locally.

## Concurrency
casd does not use any locking mechanism for reads and writes. To guarantee concurrency, writes are performed using an intermediate temporary file:

1. Create a temporary file using `mkdtemp()` (which is an atomic operation)
2. Write the contents to it (the current process has exclusive ownership of it)
3. Create a hard link from the final destination to the temporary file (also an atomic operation)
4. `unlink()` the temporary file

This works because of the guarantee that—because we write to files with their digests as names—the contents to be written to a file are going to be the same across all processes.

Because hard links can only link two entries in the same filesystem, intermediate files are created under `CAS_PATH/tmp`.

## ``disk_usage`` metadata file
To reduce start up times when the size of the cache is large, casd will write a metadata file containing the disk usage at the instant it shuts down.

That file, ``CAS_PATH/cas/disk_usage``, will enable subsequent instances of casd to avoid computing the disk usage at start time.

Like the rest of the CAS directory, the metadata file can be read but must not be modified. (Note that casd will delete the file after the first operation that alters disk usage. It won't be created until casd exits gracefully, and therefore is not suitable for live monitoring.)

## Expiry
In order to limit the amount of disk space used, casd provides an [expiry mechanism](https://gitlab.com/BuildGrid/buildbox/buildbox-casd/issues/6) based on the LRU (*Least Recently Used*) algorithm.

The mechanism (disabled by default) can be configured with two values: a low-watermark and a high-watermark. When the disk usage exceeds the value of the high-watermark, the cleanup process is triggered. Data is then deleted, starting from the oldest entries, until the size in use falls under the low-watermark.

That mechanism relies only on the `mtime` value provided by the filesystem to determine the age of an entry and it is *blob-based*: deletes blobs with the oldest timestamps without taking references into account.

(The amount of storage in use is tracked internally by a variable of type `std::atomic<int64_t>`, which is suitable since casd is the only process that can write files into the storage.)

### Available space monitoring
casd will also regularly (every 5 seconds) monitor the total disk usage to make sure that the high-watermark is not excessive compared to the actual free space. If that were to happen, casd could fail to expire blobs before the disk gets full.

If necessary, that mechanism will reduce the high-watermark so that it is equal to the current cache usage plus the available space minus some configurable headroom. It will also adjust the low-watermark so that the same high:low watermark ratio is maintained.

The current space used by the local cache as well as the value of an optional quota can be queried using the `GetLocalDiskUsage()` function of the LocalCAS protocol.
