What is ``buildbox-casd``?
==========================

``buildbox-casd`` is a long-lived local cache and proxy for the
``ContentAddressableStorage`` services described by Bazel's
`Remote Execution API`_. Its intended purpose is to run on the same host
as a remote worker or client to provide fast access to data in CAS.

.. _Remote Execution API: https://docs.google.com/document/d/1AaGk7fOPByEvpAbqeXIyE8HX_A3_axxNnvroblTZ_6s

It also supports the `LocalCAS Protocol`_.

.. _LocalCAS Protocol: https://gitlab.com/BuildGrid/buildbox/buildbox/blob/master/protos/build/buildgrid/local_cas.proto

Usage
=====

``buildbox-casd`` can be configured in two modes:

* As a regular CAS server that stores blobs on disk. In this mode, ``buildbox-casd`` is the source of truth for all blobs in CAS.

* As a CAS proxy. In this mode ``buildbox-casd`` is pointed to a remote CAS server, and it will forward all requests to it while caching data locally.

   buildbox-casd [OPTIONS] LOCAL_CACHE_PATH

This starts a daemon storing local cached data at the ``LOCAL_CACHE_PATH`` path.

The key parameters are:

* ``--remote=URL``

  URL of both remote CAS and Remote Asset servers, may be overriden by specifying
  ``--cas-remote`` or ``--ra-remote`` (see below).

* ``--cas-remote=URL``

  URL of the remote CAS server to which requests should be forwarded to.
  Setting this option configures ``buildbox-casd`` to act in proxy mode.

  If this is not provided, ``buildbox-casd`` behaves like a regular CAS server storing blobs on disk.

  If this is provided, an additional server instance can be created with ``--server-instance=NAME``.

* ``--ra-remote=URL``

  URL of the Remote Asset server to which requests should be forwarded to
  Setting this option configures ``buildbox-casd`` to act in the proxy mode.

  If this is not provided, ``buildbox-casd`` behaves like a regular Remote Asset server storing blobs on disk.

* ``--proxy-instance=NAME``

  Instance used for connecting to CAS and Remote Asset servers.

* ``--local-server-instance=NAME``

  Name given to the new LocalCAS instance (if not set, an empty string).

* ``--verbose``

  Output additional information to stdout that may be useful when debugging.

* ``--bind=URL``

  Endpoint where ``casd`` listens for incoming requests.

  Multiple addresses can be specified.

  By default a Unix domain socket in ``LOCAL_CACHE_PATH/casd.sock``.

* ``--metrics-mode=MODE``

  Where mode is one of the following: ``udp://<hostname>:<port>``, ``file:///path/to/file``, or ``stderr``

* ``--metrics-publish-interval=VALUE``

  Where VALUE is the number of seconds that the background thread sleeps in between publishing metrics


Full usage
----------
::

   --help                           Display usage and exit.
   --version                        Print version information and exit. [optional]
   --remote                         URL for all services [optional]
   --instance                       Instance for all services [optional]
   --server-cert                    Server TLS certificate for all services (PEM-encoded) [optional]
   --client-key                     Client private TLS key far all services (PEM-encoded) [optional]
   --client-cert                    Client TLS certificate for all services (PEM-encoded) [optional]
   --access-token                   Authentication token for all services (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
   --token-reload-interval          Default access token refresh timeout [optional]
   --googleapi-auth                 Use GoogleAPIAuth for all services [optional]
   --retry-limit                    Retry limit for gRPC errors for all services [optional]
   --retry-delay                    Retry delay for gRPC errors for all services [optional]
   --retry-on-code                  gRPC status code(s) as string(s) to retry on for all services, e.g. 'UNKNOWN', 'INTERNAL' [optional]
   --request-timeout                Timeout for gRPC requests for all services  (set to 0 to disable timeout) [optional]
   --min-throughput                 Minimum throughput for gRPC requests for all services, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
   --keepalive-time                 gRPC keepalive pings period for all services (set to 0 to disable keepalive pings) [optional]
   --load-balancing-policy          gRPC load balancing policy for all services (valid options are 'round_robin' and 'grpclb') [optional]
   --cas-remote                     URL for the CAS service [optional]
   --cas-instance                   Name of the CAS instance [optional]
   --cas-server-cert                Server TLS certificate for CAS (PEM-encoded) [optional]
   --cas-client-key                 Client private TLS key for CAS (PEM-encoded) [optional]
   --cas-client-cert                Client TLS certificate for CAS (PEM-encoded) [optional]
   --cas-access-token               Authentication token for CAS (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
   --cas-token-reload-interval      Access token refresh timeout for CAS service [optional]
   --cas-googleapi-auth             Use GoogleAPIAuth for CAS service [optional]
   --cas-retry-limit                Retry limit for gRPC errors for CAS service [optional]
   --cas-retry-delay                Retry delay for gRPC errors for CAS service [optional]
   --cas-retry-on-code              gRPC status code(s) as string(s) to retry on for CAS service e.g., 'UNKNOWN', 'INTERNAL' [optional]
   --cas-request-timeout            Timeout for gRPC requests for CAS service (set to 0 to disable timeout) [optional]
   --cas-min-throughput             Minimum throughput for gRPC requests for CAS service, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
   --cas-keepalive-time             gRPC keepalive pings period for CAS service (set to 0 to disable keepalive pings) [optional]
   --cas-load-balancing-policy      gRPC load balancing policy for CAS service (valid options are 'round_robin' and 'grpclb') [optional]
   --server-instance                [Deprecated] Use --local-server-instance. [optional]
   --local-server-instance          Create an additional local CAS server instance (useful when --cas-remote is specified). [optional]
   --digest-function                Set a custom digest function. Default: SHA256
                                       Supported functions: SHA384, SHA512, SHA256, SHA1, MD5 [optional]
   --ac-remote                      URL for the Action Cache service [optional]
   --ac-instance                    Name of the Action Cache instance [optional]
   --ac-server-cert                 Server TLS certificate for Action Cache (PEM-encoded) [optional]
   --ac-client-key                  Client private TLS key for Action Cache (PEM-encoded) [optional]
   --ac-client-cert                 Client TLS certificate for Action Cache (PEM-encoded) [optional]
   --ac-access-token                Authentication token for Action Cache (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
   --ac-token-reload-interval       Access token refresh timeout for Action Cache service [optional]
   --ac-googleapi-auth              Use GoogleAPIAuth for Action Cache service [optional]
   --ac-retry-limit                 Retry limit for gRPC errors for Action Cache service [optional]
   --ac-retry-delay                 Retry delay for gRPC errors for Action Cache service [optional]
   --ac-retry-on-code               gRPC status code(s) as string(s) to retry on for Action Cache service e.g., 'UNKNOWN', 'INTERNAL' [optional]
   --ac-request-timeout             Timeout for gRPC requests for Action Cache service (set to 0 to disable timeout) [optional]
   --ac-min-throughput              Minimum throughput for gRPC requests for Action Cache service, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
   --ac-keepalive-time              gRPC keepalive pings period for Action Cache service (set to 0 to disable keepalive pings) [optional]
   --ac-load-balancing-policy       gRPC load balancing policy for Action Cache service (valid options are 'round_robin' and 'grpclb') [optional]
   --ra-remote                      URL for the Remote Asset service [optional]
   --ra-instance                    Name of the Remote Asset instance [optional]
   --ra-server-cert                 Server TLS certificate for Remote Asset (PEM-encoded) [optional]
   --ra-client-key                  Client private TLS key for Remote Asset (PEM-encoded) [optional]
   --ra-client-cert                 Client TLS certificate for Remote Asset (PEM-encoded) [optional]
   --ra-access-token                Authentication token for Remote Asset (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
   --ra-token-reload-interval       Access token refresh timeout for Remote Asset service [optional]
   --ra-googleapi-auth              Use GoogleAPIAuth for Remote Asset service [optional]
   --ra-retry-limit                 Retry limit for gRPC errors for Remote Asset service [optional]
   --ra-retry-delay                 Retry delay for gRPC errors for Remote Asset service [optional]
   --ra-retry-on-code               gRPC status code(s) as string(s) to retry on for Remote Asset service e.g., 'UNKNOWN', 'INTERNAL' [optional]
   --ra-request-timeout             Timeout for gRPC requests for Remote Asset service (set to 0 to disable timeout) [optional]
   --ra-min-throughput              Minimum throughput for gRPC requests for Remote Asset service, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
   --ra-keepalive-time              gRPC keepalive pings period for Remote Asset service (set to 0 to disable keepalive pings) [optional]
   --ra-load-balancing-policy       gRPC load balancing policy for Remote Asset service (valid options are 'round_robin' and 'grpclb') [optional]
   --exec-remote                    URL for the Execution service [optional]
   --exec-instance                  Name of the Execution instance [optional]
   --exec-server-cert               Server TLS certificate for Execution (PEM-encoded) [optional]
   --exec-client-key                Client private TLS key for Execution (PEM-encoded) [optional]
   --exec-client-cert               Client TLS certificate for Execution (PEM-encoded) [optional]
   --exec-access-token              Authentication token for Execution (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
   --exec-token-reload-interval     Access token refresh timeout for Execution service [optional]
   --exec-googleapi-auth            Use GoogleAPIAuth for Execution service [optional]
   --exec-retry-limit               Retry limit for gRPC errors for Execution service [optional]
   --exec-retry-delay               Retry delay for gRPC errors for Execution service [optional]
   --exec-retry-on-code             gRPC status code(s) as string(s) to retry on for Execution service e.g., 'UNKNOWN', 'INTERNAL' [optional]
   --exec-request-timeout           Timeout for gRPC requests for Execution service (set to 0 to disable timeout) [optional]
   --exec-min-throughput            Minimum throughput for gRPC requests for Execution service, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
   --exec-keepalive-time            gRPC keepalive pings period for Execution service (set to 0 to disable keepalive pings) [optional]
   --exec-load-balancing-policy     gRPC load balancing policy for Execution service (valid options are 'round_robin' and 'grpclb') [optional]
   --proxy-instance                 Name of proxy instance. Multiple values of PROXY_INSTANCE[:REMOTE_INSTANCE] can be specified. If REMOTE_INSTANCE exists, the proxy instance uses it for remote connections, overriding --*instance options. [optional]
   --bind                           Bind to address:port or UNIX socket in unix:path. Multiple addresses can be specified. [optional]
   --unix-socket-group              If binding to a UNIX socket, update its group ownership [optional]
   --unix-socket-mode               If binding to a UNIX socket, update its mode in octal format e.g. 0775 [optional]
   --permit-keepalive-time          The minimum allowed time in seconds between keepalive pings from clients [optional]
   --quota-high                     Maximum local cache size (e.g., 50G or 2T) [optional]
   --quota-low                      Local cache size to retain on LRU expiry, either as absolute size or as percentage of `quota-high` [optional, default = "50%"]
   --reserved                       Reserved disk space (headroom left when `quota-high` gets automatically reduced) [optional, default = "2G"]
   --protect-session-blobs          Do not expire blobs created or used in the current session [optional, default = false]
   --buildbox-run                   Absolute path to runner executable to enable the local execution service [optional]
   --runner-arg                     Arguments to pass buildbox-run when buildbox-casd runs a job [optional]
   --jobs                           Maximum number of jobs to run concurrently for local execution [optional, default = 4]
   --exec-hybrid-queue-limit        Maximum queue length for local execution in proxy instance [optional]
   --exec-fallback-to-local         Fallback to local execution if the remote server is unable to schedule the clients request (e.g. due to service throttling). Fallback executions are excluded from the --exec-hybrid-queue-limit value. Requires that --buildbox-run is set. [optional, default = false]
   --cache-failures                 Cache local actions with non-zero exit code [optional, default = true]
   --findmissingblobs-cache-ttl     Number of seconds to cache the results of
                                       FindMissingBlobs() calls when in proxy mode [optional, default = 0]
   --capture-allow-file-moves       Attempt to move files into the CAS when a LocalCAS capture request has its `move_files` flag set. This will avoid a copy only when a file is in the same filesystem as the LOCAL_CACHE directory and owned by the user that is running casd. NOTE: Only enable if clients can be trusted to not break the consistency of the storage. [optional, default = false]
   --metrics-mode                   Metrics Mode: --metrics-mode=MODE - options for MODE are
                                       udp://<hostname>:<port>
                                       file:///path/to/file
                                       stderr [optional]
   --metrics-publish-interval       Metrics publishing interval [optional, default = 15]
   --metrics-prefix                 Prefix of metric names [optional, default = "buildboxcasd"]
   --read-only-remote               Specifies if casd should update the remote CAS with local build artifacts [optional]
   --store-writable-objects         Store object files with writable mode (644) [optional]
   --num-digest-threads             The number of threads to hash files to digests. Default=min(hardware-concurrency,16) [optional]
   --num-io-threads                 The number of threads for proxy CAS instances to upload or download blobs from/to remote CAS. For now, only upload is parallelized.Default=min(hardware-concurrency*2,32) [optional]
   --shutdown-deadline              Forcefully terminate pending requests the specified number of seconds after requesting shutdown (SIGINT/SIGTERM) [optional]
   --allow-pre-unstage-command      A list of _absolute paths_ to executables that are allowed to run before staged directories are unstaged. [optional]
   --ac-validation-cache-ttl        Number of seconds to cache the results of GetActionResult() validations when in proxy mode [optional, default = 0]
   --log-level                      Log level (debug, error, info, trace, warning) [optional, default = "error"]
   --verbose                        Set log level to 'debug' [optional]
     Cache path                     POSITIONAL [required]


**Note**: URLs can be of the form ``http(s)://address:port``, ``unix:path`` (with a relative or absolute path), or ``unix://path`` (with an absolute path).

Staging and Capturing
=====================

As mentioned in the introduction, ``buildbox-casd`` supports the `LocalCAS Protocol`_.
This allows you to put files and directories from the remote filesystem onto
your local filesystem (known as "Staging") and snapshot files or a directory structure
for placement into CAS (known as "Capturing"). Using this protocol allows you to
effectively use buildbox-casd as a local filesystem cache for your remote CAS.

`buildbox-worker`_ supports communication with this protocol. To enable usage, pass
:code:`--runner-arg=--use-localcas` on the **buildbox-worker** commandline.

The stage/capture mechanism can be used in two modes: with FUSE and with hardlinks.
The FUSE mode requires `buildbox-fuse`_ to be present on the system, NodeProperties are
supported, and writing to the input files is generally safe. The hardlinks mode does not
require the ``buildbox-fuse`` tool, but NodeProperties are not supported, and writing to
staged files within actions is unsafe since future actions will read the modified files.
FUSE is enabled by default, and if ``buildbox-fuse`` is not on the system or FUSE is not
supported on the system, ``buildbox-casd`` will use hardlinks.

**For staging and capturing to be completely safe in hardlinks mode, please run**
**buildbox-casd as a different user from the client.** If you run them as the same user,
actions sent to the worker in hardlinks mode may corrupt the staged files.

.. _buildbox-worker: https://gitlab.com/BuildGrid/buildbox/-/blob/master/worker

.. _LocalCAS Protocol: https://gitlab.com/BuildGrid/buildbox/buildbox/blob/master/protos/build/buildgrid/local_cas.proto

.. _buildbox-fuse: https://gitlab.com/BuildGrid/buildbox/-/blob/master/fuse

Metrics
=======
`buildbox-casd` publishes `statsd`_  runtime metrics throughout its lifetime. Documentation listing the names and descriptions of these published metrics can be found with the `MetricName`_ definitions.

.. _statsd: https://github.com/statsd/statsd/blob/master/docs/metric_types.md
.. _MetricName: https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/casd/buildboxcasd_metricnames.cpp

Metrics Enablement
------------------
To enable metrics publishing, two command line options are required:
::

        --metrics-mode=MODE                Options for MODE are:
           udp://<hostname>:<port>
           file:///path/to/file
           stderr

        --metrics-publish-interval=VALUE   Publish metric at the specified interval rate in seconds, defaults 15 seconds

Example #1: Write metrics to a statsd server on the local host listening on port 50051 and configure the background publishing thread to publish every 5 seconds
::
        --metrics-mode=udp://localhost:50051 --metrics-publish-interval=5

Example #2: Write metrics to stderr and configure the background publishing thread to publish every 5 seconds
::
        --metrics-mode=stderr --metrics-publish-interval=5

Example #3: Write metrics to a file and configure the background publishing thread to publish every 5 seconds
::
        --metrics-mode=file:///tmp/my-metrics.log --metrics-publish-interval=5
