What is ``buildbox-worker``?
============================

``buildbox-worker`` is a program that accepts jobs from a build server, invokes
a ``buildbox-run`` command to run them, and sends the results back. It
implements Bazel's `Remote Workers API`_.

It is designed to work with `BuildGrid`_ but does not depend on it.

.. _Remote Workers API: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU
.. _BuildGrid: https://buildgrid.build


::

      Usage: ./buildbox-worker
         --help                                Display usage and exit.
         --version                             Print version information and exit. [optional]
         --instance                            Instance name to pass to the Remote Workers API and the CAS servers [optional, default = ""]
         --digest-function                     Set a custom digest function. Supported functions: SHA384, SHA512, SHA256, SHA1, MD5 [optional, default = "SHA256"]
         --concurrent-jobs                     Number of jobs to run at once, defaults to running one job at a time [optional, default = 1]
         --consecutive-errors-threshold        Mark worker as UNHEALTHY after N consecutive errors, disabled by default (0 = disabled) [optional, default = 0]
         --priority                            Worker priority for job scheduling. A lower value means that the worker should be more prioritized than workers having a greater priority value. [optional, default = 0]
         --disk-usage-health-check             Mark worker as UNHEALTHY if disk usage exceeds threshold.
                                               Format: --disk-usage-health-check=/path:80%
                                               Can be specified multiple times for different paths. [optional]
         --stop-after                          Terminate after running the given number of jobs, defaults to non-terminating [optional, default = -1]
         --remote                              URL for all services [optional]
         --instance                            Instance for all services [optional]
         --server-cert                         Server TLS certificate for all services (PEM-encoded) [optional]
         --client-key                          Client private TLS key far all services (PEM-encoded) [optional]
         --client-cert                         Client TLS certificate for all services (PEM-encoded) [optional]
         --access-token                        Authentication token for all services (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
         --token-reload-interval               Default access token refresh timeout [optional]
         --googleapi-auth                      Use GoogleAPIAuth for all services [optional]
         --retry-limit                         Retry limit for gRPC errors for all services [optional]
         --retry-delay                         Retry delay for gRPC errors for all services [optional]
         --retry-on-code                       gRPC status code(s) as string(s) to retry on for all services, e.g. 'UNKNOWN', 'INTERNAL' [optional]
         --request-timeout                     Timeout for gRPC requests for all services  (set to 0 to disable timeout) [optional]
         --min-throughput                      Minimum throughput for gRPC requests for all services, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
         --keepalive-time                      gRPC keepalive pings period for all services (set to 0 to disable keepalive pings) [optional]
         --load-balancing-policy               gRPC load balancing policy for all services (valid options are 'round_robin' and 'grpclb') [optional]
         --bots-remote                         URL for the Bots service [optional]
         --bots-instance                       Name of the Bots instance [optional]
         --bots-server-cert                    Server TLS certificate for Bots (PEM-encoded) [optional]
         --bots-client-key                     Client private TLS key for Bots (PEM-encoded) [optional]
         --bots-client-cert                    Client TLS certificate for Bots (PEM-encoded) [optional]
         --bots-access-token                   Authentication token for Bots (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
         --bots-token-reload-interval          Access token refresh timeout for Bots service [optional]
         --bots-googleapi-auth                 Use GoogleAPIAuth for Bots service [optional]
         --bots-retry-limit                    Retry limit for gRPC errors for Bots service [optional]
         --bots-retry-delay                    Retry delay for gRPC errors for Bots service [optional]
         --bots-retry-on-code                  gRPC status code(s) as string(s) to retry on for Bots service e.g., 'UNKNOWN', 'INTERNAL' [optional]
         --bots-request-timeout                Timeout for gRPC requests for Bots service (set to 0 to disable timeout) [optional]
         --bots-min-throughput                 Minimum throughput for gRPC requests for Bots service, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
         --bots-keepalive-time                 gRPC keepalive pings period for Bots service (set to 0 to disable keepalive pings) [optional]
         --bots-load-balancing-policy          gRPC load balancing policy for Bots service (valid options are 'round_robin' and 'grpclb') [optional]
         --cas-remote                          URL for the CAS service [optional]
         --cas-instance                        Name of the CAS instance [optional]
         --cas-server-cert                     Server TLS certificate for CAS (PEM-encoded) [optional]
         --cas-client-key                      Client private TLS key for CAS (PEM-encoded) [optional]
         --cas-client-cert                     Client TLS certificate for CAS (PEM-encoded) [optional]
         --cas-access-token                    Authentication token for CAS (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
         --cas-token-reload-interval           Access token refresh timeout for CAS service [optional]
         --cas-googleapi-auth                  Use GoogleAPIAuth for CAS service [optional]
         --cas-retry-limit                     Retry limit for gRPC errors for CAS service [optional]
         --cas-retry-delay                     Retry delay for gRPC errors for CAS service [optional]
         --cas-retry-on-code                   gRPC status code(s) as string(s) to retry on for CAS service e.g., 'UNKNOWN', 'INTERNAL' [optional]
         --cas-request-timeout                 Timeout for gRPC requests for CAS service (set to 0 to disable timeout) [optional]
         --cas-min-throughput                  Minimum throughput for gRPC requests for CAS service, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
         --cas-keepalive-time                  gRPC keepalive pings period for CAS service (set to 0 to disable keepalive pings) [optional]
         --cas-load-balancing-policy           gRPC load balancing policy for CAS service (valid options are 'round_robin' and 'grpclb') [optional]
         --logstream-remote                    URL for the LogStream service [optional]
         --logstream-instance                  Name of the LogStream instance [optional]
         --logstream-server-cert               Server TLS certificate for LogStream (PEM-encoded) [optional]
         --logstream-client-key                Client private TLS key for LogStream (PEM-encoded) [optional]
         --logstream-client-cert               Client TLS certificate for LogStream (PEM-encoded) [optional]
         --logstream-access-token              Authentication token for LogStream (JWT, OAuth token etc), will be included as an HTTP  Authorization bearer token [optional]
         --logstream-token-reload-interval     Access token refresh timeout for LogStream service [optional]
         --logstream-googleapi-auth            Use GoogleAPIAuth for LogStream service [optional]
         --logstream-retry-limit               Retry limit for gRPC errors for LogStream service [optional]
         --logstream-retry-delay               Retry delay for gRPC errors for LogStream service [optional]
         --logstream-retry-on-code             gRPC status code(s) as string(s) to retry on for LogStream service e.g., 'UNKNOWN', 'INTERNAL' [optional]
         --logstream-request-timeout           Timeout for gRPC requests for LogStream service (set to 0 to disable timeout) [optional]
         --logstream-min-throughput            Minimum throughput for gRPC requests for LogStream service, bytes per seconds. The value may be suffixed with K, M, G or T. [optional]
         --logstream-keepalive-time            gRPC keepalive pings period for LogStream service (set to 0 to disable keepalive pings) [optional]
         --logstream-load-balancing-policy     gRPC load balancing policy for LogStream service (valid options are 'round_robin' and 'grpclb') [optional]
         --max-wait-time                       Sets the maximum amount of time in seconds between worker polling the Bots service. [optional, default = 60]
         --graceful-shutdown-timeout           Sets the amount of time in seconds to wait for ongoing executions during a graceful shutdown signal, forcibly terminating on timeout.
                                                If not provided, the worker will wait for all executions to complete during a graceful shutdown.
                                                Format: --graceful-shutdown-timeout=TIMEOUT where TIMEOUT > 0 [optional]
         --preserve-update-bot-session         When an execution is completed, preserve the previous UpdateBotSession RPC instead of cancelling it.
                                                If not provided, the previous RPC will be cancelled to send fresh state sooner. [optional, default = false]
         --runner-use-job-instance             When set, parse the lease ID at the last '/' separator and use the prefix as the
                                                instance name for runner CAS and asset connections. [optional, default = false]
         --buildbox-run                        Absolute path to runner executable [required]
         --runner-arg                          Arguments to pass buildbox-run when the worker runs a job
                                                This can be useful if the buildbox-run implementation you're using supports non-standard options [optional]
         --platform                            Add a key-value pair to the 'Platform' message the worker sends to the server
                                                (see the Remote Workers API specification for keys and values you can use)
                                                format is --platform KEY=VALUE [optional]
         --metrics-mode                        format is --metrics-mode=MODE where 'MODE' is one of:
                                                udp://<hostname>:<port>
                                                file:///path/to/file
                                                stderr [optional]
         --metrics-publish-interval            Metrics publishing interval [optional]
         --metrics-statsd-format               Metrics statsd format [optional, default = "influx"]
         --log-level                           Log verbosity level [optional, default = "info"]
         --verbose                             Set log level to debug [optional]
         --log-directory                       Write logs to this directory with filenames '<program name>.<hostname>.<user name>.log.<severity level>.<date>.<time>.<pid>' [optional, default = ""]
         --config-file                         Absolute path to config file [optional, default = ""]
         BOT Id                              POSITIONAL [optional]

The ``buildbox-run`` interface
==============================

``buildbox-worker`` invokes a ``buildbox-run`` command to download a job's
files, run the job, and upload the results. You can change the command with
the ``--buildbox-run=BUILDBOXRUN`` option to change the fetching and
sandboxing mechanisms the worker uses.

The command you specify must support the following interface:

  BUILDBOXRUN --remote=URL [--instance=NAME] [--server-cert=CERT] [--server-key=KEY]
  [--client-key=KEY] --action=FILE --action-result=FILE

``--remote`` and  ``--instance`` (empty by default) specify the CAS server.
``--server-cert``, ``--server-key``, and ``--client-key`` work as they do in
``buildbox-worker``.  ``--action`` specifies the path to a file with a
serialized Action protocol buffer, and ``--action-result`` is the path to write
the corresponding ActionResult to.


The ``buildbox-worker configuration`` file
==============================

``buildbox-worker`` supports a protobuf-based configuration enabled via the
``--config-file=FILE`` parameter, where 'FILE' points to the absolute path of
the file. This file currently only supports a 'botStatus' entry. Format of the
file is:

botStatus: [0-4]

To be backwards compatible, if no ``--config-file=FILE`` is on the command-line,
the worker will set it's initial status to ``OK``.

Otherwise, the worker will read in the status from the config file and use that
as the initial state to be passed to a bots service.

Metrics
=======
buildbox-worker publishes runtime metrics throughout its lifetime. Documentation listing the names of these published metrics can be found in `buildboxworker_metricnames.cpp`_ and a brief description of them can be found in the `buildboxworker_metricnames.h`_ header file.

.. _buildboxworker_metricnames.cpp: https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/worker/buildboxworker_metricnames.cpp
.. _buildboxworker_metricnames.h: https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/worker/buildboxworker_metricnames.h

Metrics Enablement
------------------
To enable metrics publishing, two command line options are required:
::

   --metrics-mode=MODE                Options for MODE are:
           udp://<hostname>:<port>
           file:///path/to/file
           stderr

   --metrics-publish-interval=VALUE   Publish metric at the specified interval rate in seconds, defaults 15 seconds

In addition to the above options, the ``--metrics-statsd-format`` option can be used to specify the format of the metrics when using the statsd publisher.
Supported formats are 'influx', 'graphite' and 'dog', with 'influx' being the default.

Example #1: Write metrics to a statsd server on the local host listening on port 50051 and configure the background publishing thread to publish every 5 seconds
::
   --metrics-mode=udp://localhost:50051 --metrics-publish-interval=5

Example #2: Write metrics to stderr and configure the background publishing thread to publish every 5 seconds
::
   --metrics-mode=stderr --metrics-publish-interval=5

Example #3: Write metrics to a file and configure the background publishing thread to publish every 5 seconds
::
   --metrics-mode=file:///tmp/my-metrics.log --metrics-publish-interval=5

Stdout/stderr streaming (LogStream API)
=======================================

buildbox-worker supports using the `LogStream API`_ to stream the contents that a command writes to its standard outputs in real time.

To request that, a remote execution server can attach an ``ExecuteOperationMetadata`` message with key ``"executeoperationmetadata-bin"`` to the `trailing metadata`_ returned by the ``UpdateBotSession()`` method.

That metadata message must contain the digest of the action to be streamed and a LogStream write resource name in its ``stdout_stream_name`` field. The worker will stream stdout and stderr merged in the same stream, so the output will be equivalent to the one observed if the command was executing in a local terminal.


Debugging/testing
-----------------
To enable running automated tests and help debugging the streaming mechanism, buildbox-worker provides an optional ``--launch-logstream-command`` CLI option to specify a command that will be executed before invoking the runner. Occurrences of ``{stdout}`` and ``{stderr}`` in the string get replaced with the respective resource names that were issued by the execution server (note that the names belong to the write end of the LogStreams, and they contain a token that is private to the worker).

The runner will not be invoked until the command returns.

Note that this option is only intended for testing purposes. To enable it, add options ``-DCMAKE_BUILD_TYPE=DEBUG -DLOGSTREAM_DEBUG=ON`` to cmake when building buildbox-worker.

.. _LogStream API: https://groups.google.com/g/remote-execution-apis/c/LCLsBSgSnU0/m/yTTtqr-cAwAJ
.. _trailing metadata: https://grpc.github.io/grpc/cpp/classgrpc__impl_1_1_client_context.html#a81d5df78eb77fb0d1e97220f656afd95
