# C implementation of RFC6330 RaptorQ Codes for Librecast

![Librecast Logo](https://www.librecast.net/media/librecast-logo_dark.svg)

IP  Multicast  is based on UDP, which is inherently unreliable. Packets may
arrive out of order, or not at all. TCP provides unicast with a reliable
messaging layer on top of this unreliable, connectionless medium.

Unicast,  however, is one-to-one only. Multicast could, in theory, use all of
the same reliability options (ACKs etc.) as TCP, at the cost of not being
scalable any more.

Fortunately there are other ways to achieve similar reliability.  RFC3208
describes Pragmatic General Multicast (PGM) based on NAKs (negative
acknowledgements). This, too, has scaling issues.

Forwards Error Correction (FEC) offers us another approach.

Thanks to parity checking in the network stack, we don't generally need to worry
about errors within packets. Every packet has a checksum, and if that doesn't
match, the packet is dropped before it reaches us. Our encryption provides
further checking of  data  received.   We  need only concern ourselves with
erasures.  ie. dropped packets.

RaptorQ is an implementation of a class of systematic erasure codes called
fountain codes.

The  data we want to send is split into blocks, and then pre-encoded into a set
of intermediate symbols.  From these intermediate symbols we can generate both
our original  source  symbols, and also additional repair symbols.

Provided  the  recipient  receives  at least a minimum value K' of these
symbols (any unique combination of source and repair) the intermediate symbols
can  be  reconstituted,  and  the original data recovered.

RaptorQ  is what is called a systematic encoding, because the set of symbols we
send includes our original data as plain text. Provided all source symbols are
received, the original  data has been transmitted with no decoding overhead.  It
is only in the case where we need to supplement the source symbols with repair
symbols that we must perform the decoding process.

## Documentation

See the man pages:

- `lcrq(7)`
- `rq_init(3)`
- `rq_free(3)`
- `rq_query(3)`
- `rq_encode(3)`
- `rq_decode(3)`
- `rq_symbol(3)`
- `lcrq-scatter(1)`
- `lcrq-gather(1)`
- `lcrq-params(1)`
- `lcrq-speedtest(1)`

## Installation

Oh good. You're one of those people who reads documentation. Phew! Please, do
NOT just build the sources with the usual `configure && make`. It will run like a
dog with no legs.

C convention says that I absolutely MUST NOT mess with CFLAGS and try to help
you in any way by setting sane defaults.  That means you need to do it.

How you build LCRQ depends on whether you are building it to just run on your
local machine, or if you are building binaries that need to support multiple
CPUs, such as when building packages for a binary distribution.

There should be no compiler warnings, even with `-Wall -Wextra -pedantic`.
Please raise a bug if you see any.

### Supported Compiliers

The code compiles and has been tested with gcc and clang.  The default is
whatever your default CC is, which you can set when calling `configure`. eg.
`./configure CC=clang`.

### Configure Options

#### Native Compilation (best performance)

LCRQ makes heavy use of SIMD instructions when available. For best performance,
a native build targeting your local machine is best.

If you are compiling lcrq for use only on your local machine, you will want to
set some CFLAGS appropriate to your platform. You can read more about these
options in the [GCC manual](https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html).

Consider:

`-O3`
: Optimize aggressively. Highly recommended.

`-march=native`
: Enable all CPU instruction sets available on the local machine, and tuning for
your CPU. Combine this with `--enable-native` for best results.

`-mpopcnt`
: In theory this will be enabled by `-march=native`, but it doesn't hurt to be
explicit if your CPU supports it.

`--enable-native`
: Enable native optimizations in the LCRQ build. Combine supported instruction
sets into one compilation unit, rather than building separately. NB: Requires that
your CPU supports AVX2.

`-flto`
: Enable link-time optimization to allow compiler optimization across
compilation units. Generally safe to use and highly recommended.

`-funroll-loops`
: Give the compiler a bit more encouragement to unroll loops.

`-ffast-math`
: Recommended, but might be unsafe in some circumstances. Try it. If you have
problems, turn it off.

`-DNDEBUG`
: Disable assertions and other debugging code.

So, putting that all together, we get something like:

```
./configure CFLAGS="-Wall -Wextra -pedantic -O3 -march=native -mpopcnt -ffast-math -funroll-loops -flto -DNDEBUG" --enable-native
make
make test      # make sure all tests PASS
make install
```

A rough indication of encoding and decoding performance can be garnered by
running  `make speedtest`. This is useful for observing the effects of various
optimizations. It is not a proper benchmark test.

NB: if rebuilding with different compiler options, don't forget to run
`make clean` first.

#### Example Programs

Some example programs are available in `examples/` which show how to use the
API. These programs can be built with `make examples` and installed with `make
install-examples`. There are man pages for each of the programs:

- lcrq-params(1)
- lcrq-scatter(1)
- lcrq-gather(1)
- lcrq-speedtest(1)

#### Portable Compilation (for binary distributions)

If you are packaging LCRQ, or otherwise building it to run on a whole family of
CPUs (like x86\_64) then some of the optimizations in the previous section
aren't applicable.  You can probably safely use:

`-O3`
: Optimize aggressively. Highly recommended.

`-flto`
: Enable link-time optimization to allow compiler optimization across
compilation units. Generally safe to use and highly recommended.

`-funroll-loops`
: Give the compiler a bit more encouragement to unroll loops.

`-ffast-math`
: Recommended, but might be unsafe in some circumstances. Try it. If you have
problems, turn it off.

`-DNDEBUG`
: Disable assertions and other debugging code.

So, putting that all together, we get something like:

```
./configure CFLAGS="-Wall -Wextra -pedantic -O3 -flto -funroll-loops -ffast-math -DNDEBUG"
make
make test      # make sure all tests PASS
make install
```

Please run `make speedtest` in your build scripts and pay attention to the
output. Does it seem reasonable for the target system? Compare it to a build on
your own machine.


### \*BSD

Users of \*BSD will need to use gmake instead of make. `gmake test` requires
bash.

We test all releases on FreeBSD, NetBSD and OpenBSD.

## Other RaptorQ Implementations

TvRQ
: https://github.com/lorinder/TvRQ (C)

nanorq
: https://github.com/sleepybishop/nanorq (C)

libRaptorQ
: https://github.com/LucaFulchir/libRaptorQ (C++)

raptorq
: https://github.com/cberner/raptorq (Rust)

## Acknowledgements

LCRQ uses the fast Galois field multiplication technique described in:

 J. S. Plank and K. M. Greenan and E. L. Miller (2013)
 ["Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions"](http://web.eecs.utk.edu/~jplank/plank/papers/FAST-2013-GF.html)

## References

- https://datatracker.ietf.org/doc/html/rfc6330

- http://www.ece.ubc.ca/~janm/Papers_RG/Shokrollahi_IT_June06.pdf

- https://www.cberner.com/2019/03/30/raptorq-rfc6330-rust-optimization/

- https://www.cberner.com/2020/10/12/building-fastest-raptorq-rfc6330-codec-rust/

- https://github.com/cberner/raptorq/blob/master/RFC6330_ERRATA.md

<hr />

## Funding

<p class="bigbreak">
This project was funded through the <a href="https://nlnet.nl/discovery">NGI Zero Discovery</a>, <a href="https://nlnet.nl/assure">NGI Assure</a> and <a href="https://nlnet.nl/core">NGI Zero Core</a> funds, established by NLnet with financial support from the European Commission's <a href="https://ngi.eu">Next Generation Internet</a> programme. *See the NLnet open calls page to <a href="https://nlnet.nl/propose/">apply for funding</a> for your project*
</p>
<p>
 <a href="https://nlnet.nl/project/Librecast-OverlayMulticast/">
      <img align="left" width="250" src="https://nlnet.nl/logo/banner.png" alt="Logo NLnet: abstract logo of four people seen from above" /> </a>
<a href="https://ngi.eu/">
      <img align="right" width="250" src="https://nlnet.nl/image/logos/NGI0Core_tag.svg" alt="NGI Core Logo" />
 </a>
</p>
