What is buildbox-run-userchroot?
===============================

``buildbox-run-userchroot`` attempts to run actions within a chroot.
``buildbox-run-userchroot`` depends on userchroot being installed on the system.
Note that userchroot is an open source tool: https://gitlab.com/BuildGrid/buildbox/userchroot,
and is seperate from generic chroot-jails like the one provided by GNU coreutils.

Usage
=====
::

    usage: ./buildbox-run-userchroot [OPTIONS]
    --help                      Print this message and exit
    --version                   Print version information and exit
    --action=PATH               Path to read input Action from
    --action-result=PATH        Path to write output ActionResult to
    --log-level=LEVEL           (default: info) Log verbosity: trace/debug/info/warning/error
    --verbose                   Set log level to debug
    --digest-function           Set digest function
    --log-directory=DIR         Write logs to this directory with filenames:
                                buildbox-run-hosttools.<hostname>.<user name>.log.<severity level>.<date>.<time>.<pid>
    --disable-localcas          Do not use LocalCAS protocol methods
    --workspace-path=PATH       Location on disk which runner will use as root when executing jobs
    --stdout-file=FILE          File to redirect the command's stdout to
    --stderr-file=FILE          File to redirect the command's stderr to
    --no-logs-capture           Do not capture and upload the contents written to stdout and stderr
    --collect-execution-stats   Gather `execution_stats.proto` metrics and attach the digest of the message
                                in CAS to `ActionResult.execution_metadata.auxiliary_metadata`
    --validate-parameters       Only check whether all the required parameters are being passed and that no
                                unknown options are given. Exits with a status code containing the result (0 if successful).
    --remote=URL                URL for CAS service
    --instance=NAME             Name of the CAS instance
    --server-cert=PATH          Public server certificate for TLS (PEM-encoded)
    --client-key=PATH           Private client key for TLS (PEM-encoded)
    --client-cert=PATH          Public client certificate for TLS (PEM-encoded)
    --access-token=PATH         Access Token for authentication (e.g. JWT, OAuth access token, etc), will be included as an HTTP Authorization bearer token.
    --token-reload-interval=MINUTES Time to wait before refreshing access token from disk again. The following suffixes can be optionally specified: M (minutes), H (hours). Value defaults to minutes if suffix not specified.
    --googleapi-auth            Use GoogleAPIAuth when this flag is set.
    --retry-limit=INT           Number of times to retry on grpc errors
    --retry-delay=MILLISECONDS  How long to wait before the first grpc retry
    --retry-on-code=INT         Which gRPC status code(s) to retry on
    --request-timeout=SECONDS   How long to wait for gRPC request responses
    --min-throughput=BPS        Minimum throughput in bytes per second. The value may be suffixed with K, M, G or T.
    --keepalive-time=SECONDS    The period after which a keepalive ping is sent
    --load-balancing-policy     Which grpc load balancing policy to use. Valid options are 'round_robin' and 'grpclb'
    --ra-remote=URL             URL for Asset service
    --ra-instance=NAME          Name of the Asset instance
    --ra-server-cert=PATH       Public server certificate for TLS (PEM-encoded)
    --ra-client-key=PATH        Private client key for TLS (PEM-encoded)
    --ra-client-cert=PATH       Public client certificate for TLS (PEM-encoded)
    --ra-access-token=PATH      Access Token for authentication (e.g. JWT, OAuth access token, etc), will be included as an HTTP Authorization bearer token.
    --ra-token-reload-interval=MINUTES Time to wait before refreshing access token from disk again. The following suffixes can be optionally specified: M (minutes), H (hours). Value defaults to minutes if suffix not specified.
    --ra-googleapi-auth         Use GoogleAPIAuth when this flag is set.
    --ra-retry-limit=INT        Number of times to retry on grpc errors
    --ra-retry-delay=MILLISECONDS How long to wait before the first grpc retry
    --ra-retry-on-code=INT      Which gRPC status code(s) to retry on
    --ra-request-timeout=SECONDS How long to wait for gRPC request responses
    --ra-min-throughput=BPS     Minimum throughput in bytes per second. The value may be suffixed with K, M, G or T.
    --ra-keepalive-time=SECONDS The period after which a keepalive ping is sent
    --ra-load-balancing-policy  Which grpc load balancing policy to use. Valid options are 'round_robin' and 'grpclb'
    --allow-executable=ALLOWED_EXECUTABLE_PATH Only commands which run this executable are allowed.
⋊> ~/p/b/buildbox-common on zchen723/config-retry-code ⨯ ./build/run-userchroot/buildbox-run-userchroot --help                                                    14:46:13

usage: ./build/run-userchroot/buildbox-run-userchroot [OPTIONS]
        --help                      Print this message and exit
        --version                   Print version information and exit
        --action=PATH               Path to read input Action from
        --action-result=PATH        Path to write output ActionResult to
        --log-level=LEVEL           (default: info) Log verbosity: trace/debug/info/warning/error
        --verbose                   Set log level to debug
        --digest-function           Set digest function
        --log-directory=DIR         Write logs to this directory with filenames:
                                    buildbox-run-userchroot.<hostname>.<user name>.log.<severity level>.<date>.<time>.<pid>
        --disable-localcas          Do not use LocalCAS protocol methods
        --workspace-path=PATH       Location on disk which runner will use as root when executing jobs
        --stdout-file=FILE          File to redirect the command's stdout to
        --stderr-file=FILE          File to redirect the command's stderr to
        --no-logs-capture           Do not capture and upload the contents written to stdout and stderr
        --collect-execution-stats   Gather `execution_stats.proto` metrics and attach the digest of the message
                                    in CAS to `ActionResult.execution_metadata.auxiliary_metadata`
        --validate-parameters       Only check whether all the required parameters are being passed and that no
                                    unknown options are given. Exits with a status code containing the result (0 if successful).
        --remote=URL                URL for CAS service
        --instance=NAME             Name of the CAS instance
        --server-cert=PATH          Public server certificate for TLS (PEM-encoded)
        --client-key=PATH           Private client key for TLS (PEM-encoded)
        --client-cert=PATH          Public client certificate for TLS (PEM-encoded)
        --access-token=PATH         Access Token for authentication (e.g. JWT, OAuth access token, etc), will be included as an HTTP Authorization bearer token.
        --token-reload-interval=MINUTES Time to wait before refreshing access token from disk again. The following suffixes can be optionally specified: M (minutes), H (hours). Value defaults to minutes if suffix not specified.
        --googleapi-auth            Use GoogleAPIAuth when this flag is set.
        --retry-limit=INT           Number of times to retry on grpc errors
        --retry-delay=MILLISECONDS  How long to wait before the first grpc retry
        --retry-on-code             Which gRPC status code(s) to retry on
        --request-timeout=SECONDS   How long to wait for gRPC request responses
        --min-throughput=BPS        Minimum throughput in bytes per second. The value may be suffixed with K, M, G or T.
        --keepalive-time=SECONDS    The period after which a keepalive ping is sent
        --load-balancing-policy     Which grpc load balancing policy to use. Valid options are 'round_robin' and 'grpclb'
        --ra-remote=URL             URL for Asset service
        --ra-instance=NAME          Name of the Asset instance
        --ra-server-cert=PATH       Public server certificate for TLS (PEM-encoded)
        --ra-client-key=PATH        Private client key for TLS (PEM-encoded)
        --ra-client-cert=PATH       Public client certificate for TLS (PEM-encoded)
        --ra-access-token=PATH      Access Token for authentication (e.g. JWT, OAuth access token, etc), will be included as an HTTP Authorization bearer token.
        --ra-token-reload-interval=MINUTES Time to wait before refreshing access token from disk again. The following suffixes can be optionally specified: M (minutes), H (hours). Value defaults to minutes if suffix not specified.
        --ra-googleapi-auth         Use GoogleAPIAuth when this flag is set.
        --ra-retry-limit=INT        Number of times to retry on grpc errors
        --ra-retry-delay=MILLISECONDS How long to wait before the first grpc retry
        --ra-retry-on-code          Which gRPC status code(s) to retry on
        --ra-request-timeout=SECONDS How long to wait for gRPC request responses
        --ra-min-throughput=BPS     Minimum throughput in bytes per second. The value may be suffixed with K, M, G or T.
        --ra-keepalive-time=SECONDS The period after which a keepalive ping is sent
        --ra-load-balancing-policy  Which grpc load balancing policy to use. Valid options are 'round_robin' and 'grpclb'
        --userchroot-bin=PATH       Path to userchroot executable. Will default to looking in PATH if not set.
        --local-only-cas-instance-name        CAS instance name used to upload blobs of merged chroot trees.
                                        This allows switching from a proxy to a server instance to prevent those blobs from
                                        being forwarded to a remote CAS (they are only needed in the local CAS for staging).
        --inject-path=[CONTENT:]HOST_PATH:CHROOT_PATH   Map a path from HOST_PATH on the host filesystem to CHROOT_PATH inside the constructed chroot. The contents of the path will be copied into the chroot. If CONTENT is set, the injection only happens if the action has a property `inject=CONTENT`
        --sudo-bin=PATH       Path to sudo executable. Will default to looking in PATH if not set.
        --find-bin=PATH       Path to find executable. Will default to looking in PATH if not set. Required to be valid if --user is specified.
        --chmod-bin=PATH       Path to chmod executable. Will default to looking in PATH if not set. Required to be valid if --user is specified.
        --user=USER       Run userchroot as another user.
        --group=GROUP       Run userchroot in another group.

Buildbox-run-userchroot prerequisites
=====================================

In order to use ``buildbox-run-userchroot``, `userchroot`_ must be built and installed.
You must then specify the path to ``userchroot`` with the ``--userchroot-bin`` flag. Note that
this flag can be specified in `buildbox-worker`_, as a ``--runner-arg`` flag.

It is also highly recommended that `buildbox-casd`_ is used along side ``buildbox-run-userchroot``.
This is because buildbox-run-userchroot will stage an entire chroot image on every execution
request. Fetching this chroot from ``CAS`` is likely to be expensive without local caching.

.. _userchroot: https://gitlab.com/BuildGrid/buildbox/userchroot
.. _buildbox-worker: https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/worker
.. _buildbox-casd: https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/casd

Building and uploading chroot images
====================================

To use ``buildbox-run-userchroot``, a chroot image must first be built and uploaded
to ``CAS``. A chroot image is simply a filesystem image, within this context. There are
many ways to build or install one, such as `debootstrap`_. However, in this tutorial,
`chrootbuilder`_ will be used. Since it is more flexible and will ensure that the chroot
is compatible with buildbox-run-userchroot.

To use ``chrootbuilder``, simply specify a docker image or container to use. That image/container's
filesystem will exported as the chroot (Note that this process will not modify the docker image or
container). In this tutorial, the `gcc`_ docker image will be used::

    docker pull gcc@sha256:1cd1f0b466eab282b6c053e739e8510e4364fa9e91f337ba15d4a73978c3674f
    chrootbuilder -e gcc@sha256:1cd1f0b466eab282b6c053e739e8510e4364fa9e91f337ba15d4a73978c3674f chroot

Once this is done, the chroot image must be uploaded to ``CAS``. This can be done using the
`casupload`_ tool, which is included in `recc`_. This command will also calculate
the root digest::

    export RECC_CAS_SERVER=$YOUR_CAS_SERVER && export RECC_INSTANCE=$YOUR_INSTANCE
    casupload chroot

The root digest for this chroot image should be
``CHROOT_DIGEST=3e74aa72ba495b43abd18a32c4692e8ed73e808f75edcc18234b6dd66385d70d/1544``.
This digest should then be specified to ``buildbox-worker`` by passing it
``--platform chrootRootDigest=${CHROOT_DIGEST}``. It should also be specified for recc, using
``export RECC_REMOTE_PLATFORM_chrootRootDigest=${CHROOT_DIGEST}``. The ``chrootRootDigest`` key word
should also be added to the buildgrid config, as one of the ``property-keys`` under ``!execution``.
Check the `reference`_ for how to do this.

Once this setup is done, the chroot will be ready to use. ``buildbox-run-userchroot`` will use the
chroot as its sandboxing environment. Make sure to run it with ``buildbox-casd`` to have good performance.

One can find more explicit details on how to upload and use a chroot, along with ``buildbox-run-userchroot``
and ``buildbox-casd``, by checking ``test-recc-output.sh`` in the ``buildbox-e2e` repo. It uploads an alpine image,
and tests an execution request with ``buildbox-run-userchroot`` and ``buildbox-casd``.

.. _debootstrap: https://manpages.debian.org/stretch/debootstrap/debootstrap.8.en.html
.. _chrootbuilder: https://gitlab.com/BuildGrid/buildbox/chrootbuilder/
.. _gcc: https://github.com/docker-library/gcc
.. _casupload: https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/casupload/
.. _recc: https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/recc/
.. _reference: https://gitlab.com/BuildGrid/buildgrid/-/blob/master/buildgrid/_app/settings/reference.yml

Avoiding the propagation of working directories with a buildbox-casd proxy
--------------------------------------------------------------------------
When connected to a casd instance, it will be used to stage the directory in
which the action will be executed. For that buildbox-run-userchroot will need
to add to CAS a directory structure that consists of the inputs of the action
merged with the chroot image.

Since that directory tree is exclusive to a particular action, it could be
desirable to not have it be propagated to a remote CAS server. That can be
achieved by configuring casd with a server instance name
(``--server-instance=NAME``) and passing ``--local-only-cas-instance-name=NAME``
to buildbox-run-userchroot.


Limitation
==========

Userchroot has some limitations, for example the execution request should not modify the root directory. This
is because ``userchroot`` requires the chroot image's root to  have strict permissions. `buildbox-casd`_ should
also run as a different user than ``buildbox-run-userchroot``. If not, one risks execution requests corrupting the
local cache.

.. _buildbox-casd: https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/casd/
