# Embedding dependencies

To make bpftrace more portable, it has long supported an alpine-based musl
build, which statically compiled bpftrace resulting in no runtime linking
required.

The drawback to this approach is that LLVM libraries, even when statically
compiled, depends on symbols from libdl, and works best and most predictably
when dynamically linked to libc.

To embed everything except for libc, building LLVM and Clang from source is
supported. This allows for linking to arbitrary libc targets dynamically, which
may provide the best of both worlds between a purely static and a purely
dynamically-linked bpftrace executable.

For this reason, there is CMake support in the bpftrace project to build LLVM
and Clang from source, as these are the heaviest dependencies of bpftrace.
Other library dependencies can be obtained by most package managers reliably.

Upstream packages provided by package maintainers can't be depended on to
present all of the necessary built artifacts to statically link against LLVM and
Clang, despite the project [LLVM catering to this with its own CMake build
process](https://llvm.org/docs/CMake.html#embedding-llvm-in-your-project).

# Configuration flags

To make an semi-static executable that includes everything except libc, the
CMake option `STATIC_LIBC:BOOL=OFF` can be added. This should allow for a
bpftrace executable that can run on any system with a compatible libc.

To assist in this, new configuration flags allow to download, configure, build
and embed static libraries for all dependencies of bpftrace, rather than
relying on system libraries.

To build the necessary static LLVM or Clang libraries, `-DEMBED_LLVM:BOOL=ON`
and `-DEMBED_CLANG=ON` can be used respectively. Clang can link to system LLVM
as well, but may need to be patched.

It is possible to link bpftrace's embedded libraries with system libraries, so
linking to a distribution specific LLVM is possible. Many distributions do not
include `libclang.a`,  using `-DEMBED_LIBCLANG_ONLY=ON` will build only
`libclang.a`, allowing for linking to system LLVM and Clang libraries on Debian,
as opposed to Vanilla LLVM sources.

# Build times

To build from scratch on a modern development laptop with an 8-way parallel
build, bpftrace and its dependencies can be built in about 30 minutes.

Subsequent builds will not have this overhead, once the embedded dependencies
are built and cached, and will proceed at the comparable speeds as dynamically
linked builds.

## CI environments

Internal CI environments with any sort of caching mechanism should be able to
just cache any external project's build directories, and the overhead of
building on CI shouldn't be any different than other CI jobs.

# Embedding clang

On the alpine platform, libclang.a is generated by default and a static build is
already achieved using this distribution. This libclang dependency is only
achievable with custom builds, or by using alpine.

To get around this problem, the `embed_clang.cmake` file provides the necessary
static libraries for bpftrace to link against, by downloading LLVM from a github
tagged release and compiling it directly using custom cmake flags, based on
those provided by alpine already. This basically pulls the fairly simple
alpine-build "in-tree", using cmake flags based on it, and specifically
itemizing the libraries that will be link-time dependencies of bpftrace.

In turn, these clang libraries depend heavily on LLVM libraries, so this
necessitates having access to LLVM static libraries as well.

It is also possible to build only libclang from (mostly) vanilla sources, and
use system LLVM and Clang libraries. This is what the `EMBED_LIBCLANG_ONLY`
flag provides.

# Embedding LLVM

It is possible to avoid embedding LLVM by applying distribution specific
patches to embedded clang, so that it will be patch-compatible with the LLVM
libraries shipped with the target system.

This can save time by avoiding building LLVM altogether, but as distributions
LLVM libraries may have more dependencies and other bloat than the embedded
LLVM, this can result in a larger executable. On Ubuntu, the size of the
bpftrace executable is increased from 36MB to 75MB (as of Jan 13 2020).

This is still a desirable time-saver where space doesn't matter, such as when
doing Debug builds - it can avoid building 15GB of LLVM libraries.

## Distribution patches

The LLVM builds maintained by distributions have patches on top of the tagged
releases. It may be favorable from time-to-time to pull in these patches.

By downloading patches from an external source and writing a custom series
file, `quilt` can be used to apply arbitrary patches to embedded sources.

The `embed_helpers.cmake` file has the necessary helper functions to download
patches, add them to a patch series, and set the patch command to be used with
the ExternalProject.

These are currently used to patch the embedded libclang build to link to system
libraries on Debian.

Luckily this is actually pulling patches directly from LLVM's own build repo,
which is linked to by https://apt.llvm.org/. You can see the branch associated
with the LLVM version. This shows the patch series used to build LLVM 9, etc for
Debian.

In the case of LLVM 8, to get libclang.a to build only one patch, is needed,
not the whole series. The worst possible case in adding support for a new LLVM
version is to copy the series file, and set "-p2" or "-p1" depending on if it
is LLVM itself or a subproject, and what patch level it was written at.

The workflow on a new LLVM release support linking to system libs would be to:

* Try it with the existing patch series and maybe it just works
* Examine the build failure, and grep for patches affecting these files from
those in the patch series on the LLVM 9 / 10 etc branch
* Add patches to the series file until there is a minimal one that builds.
Most of the time the patches change minor cosmetic things, no major
functionality. In this case, the patch makes it from KFreebsd to kFreebsd which
breaks a header, but it is easy to fix

# Building on CI

In order to build successfully in an environment like Travis CI, it is
necessary that build jobs complete in less than 50 minutes. Unfortunately, it
takes about 50 minutes to build just LLVM dependencies alone on Travis.

In order to build a new version of LLVM, or LLVM from a cold cache, the build
must be done incrementally. This is done with controlled timeouts, to make the
build complete within the 50 minute window. This allows for progress to be
saved, and another CI job to pick up where it left off. This is done by setting
the `CI_TIMEOUT` environment variable, currently to 40 minutes (to allow time
for static bcc to build and cache artifacts to be uploaded).

It may take several successive builds to rebuild the LLVM and Clang embedded
library dependencies, when a new embedded LLVM target is added or the cache is
cleared. Once the cache is warmed, the build times should be comparable to
other builds.

## Debug build

The debug build is too large to complete successfully on Travis CI. The debug
artifacts for LLVM and Clang build process are around 35GB, which simply takes
too long to save and restore on travis, resulting in builds being killed due
to travis' 10 minute no-output timeout. This makes the incremental approach
used to warm the cache for the Release binary impracticable. 

Debug builds ultimately produce a bpftrace binary that is 1.2GB, and not
practical for redistribution. When run locally or on a less restricted CI
environment, embedded Debug build should still complete and pass all tests, but
it may take 1-2 hours for this to complete from a cold cache.

As a way to ensure that debug builds can succeed, the system LLVM may be linked
to by setting `EMBED_LLVM=OFF`.

# libc options

## glibc

This is the default, and currently only tested and supported.

bpftrace is built against the glibc from Ubuntu bionic (18.04), which provides
2.27.

## musl (not yet supported)

The original static build of bpftrace built against musl statically, but there
is no reason why it could not link against musl dynamically.

## bionic (android - not yet supported)

It should be possible to link to bionic libc for Android, allowing for bpftrace
to run on Android systems with Kernels that support it.

## uclibc (not yet supported)

Theoretically no reason this shouldn't work to dynamically link to, may be
helpful for supporting bpftrace on embedded environments.
