paivana tests
=============

This directory contains five test programs:

  reverse_proxy    an integration suite for the reverse-proxy side of
                   paivana-httpd, driven by test_reverse_proxy.sh
  paywall          an integration suite for the paywall itself, driven
                   by test_paywall.sh against a real GNU Taler system
  client_address   a unit test for the client address the access
                   cookie is keyed on (test_client_address.c)
  cookie_header    a unit test for the `Set-Cookie` line paivana emits
                   for that cookie (test_cookie_header.c)
  cookie_access    a unit test for the access decision the cookie
                   value encodes, and for the `paivana_id` the order
                   is created under (test_cookie_access.c)

The reverse-proxy suite runs paivana-httpd with `-n` (paywall
disabled) so no merchant backend is required: it only verifies that the
proxy correctly forwards HTTP requests and responses.  The paywall
suite is the other half -- everything `-n` switches off -- and needs an
exchange, a merchant backend and a bank, so it skips where those are
not installed.  Everything below describes the reverse-proxy suite
except the sections at the end.

What gets built
---------------

The test suite uses four diverse upstream HTTP server implementations
so that paivana is not exercised only against libmicrohttpd peers:

  upstream_mhd   C / libmicrohttpd  (built always)
  upstream_go    Go (net/http)      (built if `go`    is found)
  upstream_rs    Rust (std::net)    (built if `rustc` is found)
  upstream_py    Python (stdlib)    (pure interpreter; needs python3
                                     at `make check` time)

Two further, special-purpose upstreams are also built:

  early_response_upstream
                 C / raw sockets, single-connection.  Sends a 413
                 response immediately after reading the request
                 headers, BEFORE consuming the request body — used
                 to exercise paivana's handling of an early upstream
                 response that lands while the client upload is
                 still in flight.  Writes the body byte count it
                 observed to a receipt file, as a diagnostic.

                 With `--no-drain` it additionally stops reading
                 once it has answered, and parks the connection with
                 the rest of the request queued in a deliberately
                 tiny receive buffer.  That is what makes paivana's
                 outbound socket back up: measured on loopback, the
                 upstream holds ~12 KiB unread while ~240 KiB of the
                 request sits undeliverable in paivana's send buffer.
                 A proxy that waited for that write to finish before
                 acting on the response it already holds would
                 deadlock; without `--no-drain` the condition never
                 arises, because the upstream keeps reading.

  stream_upstream
                 C / raw sockets, one process per connection.  Serves
                 bodies far larger than memory, at a rate the driver
                 chooses, and in framings a conforming server library
                 will not emit: a declared `Content-Length` that is
                 not delivered, a chunked response with no terminating
                 chunk, a connection that answers and then goes silent
                 for ever.  Also reads and verifies a request body,
                 optionally answering before it has finished.

                 Bodies are a deterministic function of their own byte
                 offset rather than stored data, so a 200 MiB case
                 costs no disk on either side.  Deliberately not a
                 constant byte: a repeated character would pass a
                 comparison that duplicated or dropped a whole aligned
                 block, which is exactly the mistake a ring buffer
                 with wrong wrap arithmetic makes -- and exactly the
                 mistake this caught during development.

The four canned upstreams all implement the same endpoints (see
"Endpoints" below).  Two test clients talk to paivana directly:
`pipeline_client`, which uses BSD sockets to pipeline requests, and
`stream_client`, which verifies a body against the same generated
pattern as it arrives and reports what it saw about the *framing* --
whether the response was chunked, what `Content-Length` reached the
client, how long the first byte took relative to the last.  Verifying
incrementally is the point: a body written to a file and compared
afterwards says nothing about whether paivana streamed it or assembled
it first.

They all bind 127.0.0.1 and nothing else.  They are not hardened in
any way -- POST /echo reflects whatever body it is given and GET
/large/10485760 hands out 10 MiB per request -- and they have no
business being reachable from the network for the duration of `make
check'.  They also all reject an argument that is not a port in
1..65535 rather than defaulting: a silent fallback binds a port the
driver is not waiting for, and the failure then surfaces five seconds
later as "did not start on port NNNNN", naming the wrong thing.

Layout of the driver
--------------------

`test_reverse_proxy.sh` is the single test program automake runs.
For each available upstream (mhd / go / py / rs) it:

  1. starts the upstream on its port (see "Ports used"),
  2. starts paivana-httpd -n pointed at that upstream,
  3. runs a battery of HTTP tests with curl, wget, and the raw-socket
     pipelining client,
  4. stops paivana and moves on to the next upstream.

Cross-cutting error-path tests (405, 413, 502) are also covered, and
the final case restarts paivana pointed at a dead port to exercise
upstream-failure handling.

What each test covers
---------------------

Per-upstream battery (`run_battery`):

  GET /hello                  happy-path GET, body proxied unchanged
  GET /status/201             2xx response status is forwarded intact
  GET /status/404             4xx response status is forwarded intact
  GET /status/500             5xx response status is forwarded intact
  HEAD /hello                 HEAD method: the status is forwarded
  HEAD /large/131072          RFC 9110 section 9.3.2's one normative
                              requirement on HEAD -- "MUST NOT send
                              content in the response" -- on a path
                              that yields 128 KiB under GET, so there
                              is something to leak.  Read off the
                              socket rather than through curl, which
                              discards a body a HEAD response has no
                              business carrying and would therefore
                              report the bug as a pass.
  GET /large/131072           128 KiB response body arrives byte for
                              byte, compared against the 'A'..'Z'
                              cycle the upstreams generate rather
                              than merely counted.  (Bodies are
                              buffered whole, not streamed: a length
                              that matches says nothing about a
                              buffer reassembled in the wrong order.)
  POST /echo                  request body is forwarded unchanged;
                              body round-trip
  POST /echo (128 KiB)        the same in the request direction and
                              at a size that spans several reads:
                              128 KiB of random bytes posted and
                              compared with what comes back.  POST
                              /upload below checks only the count the
                              upstream reports, so without this
                              nothing here would notice a request
                              body that arrived complete but corrupt.
  POST /upload (64 KiB)       large random POST upload; upstream
                              reports the byte count it saw
  PUT /put                    PUT method + body forwarding
  PATCH /patch                PATCH method + body forwarding
                              (paivana sets CUSTOMREQUEST)
  DELETE /item/1              DELETE method, 204 No Content
  OPTIONS /hello              OPTIONS method, Allow header survives
                              the round-trip
  GET /echo-headers           paivana adds the reverse-proxy headers
                              X-Forwarded-For, X-Forwarded-Proto, Via
  Host: rewritten             the Host the upstream sees is the
                              authority of DESTINATION_BASE_URL, not
                              the one the client dialed, and is
                              host[:port] and nothing else (RFC 9110
                              §7.2 — no userinfo, no path, no query).
                              Note this only covers the destination
                              URLs paivana will actually accept:
                              TALER_is_web_url() rejects userinfo and
                              IPv6-literal DESTINATION_BASE_URLs at
                              startup, so those cannot be reached
                              from the driver.
  custom X-Test header        arbitrary client request headers are
                              forwarded unchanged
  X-Upstream response header  upstream response headers survive the
                              round-trip back to the client, and the
                              value names the upstream this battery
                              was pointed at -- a restart that
                              silently kept the previous destination
                              would satisfy a presence check

Forwarding-header tests (run once):

  no -f                       a client's own X-Forwarded-For /
                              -Proto / -Host must not reach the
                              upstream: paivana is the outermost
                              proxy and replaces them with what it
                              can see for itself.  In particular the
                              scheme must come from the transport,
                              not from a header the client wrote --
                              TALER_mhd_is_https() believes
                              X-Forwarded-Proto, so paivana asks MHD
                              about the TLS session instead.
  -f, chain extension         with -f paivana is behind a trusted
                              proxy: the inbound chain is preserved
                              and paivana's own peer appended to the
                              right, rather than the chain being
                              thrown away.  Also covers a repeated
                              X-Forwarded-For arriving as two field
                              lines (RFC 9110 §5.3: one combined
                              header must reach the upstream).
  -f, trusted -Proto / -Host  the values a trusted proxy sent are
                              passed through unchanged.
  unix socket                 the deployment the Debian packaging
                              ships.  A Unix peer has no address, so
                              with -f the inbound chain is forwarded
                              unadorned (nothing is appended, and no
                              placeholder is invented -- the hop is
                              recorded in Via), and without -f no
                              X-Forwarded-For is emitted at all.
                              This is the only case that reaches the
                              address-less code paths.
  RFC 7239 Forwarded          the standardized header is handled like
                              X-Forwarded-For: extended under -f,
                              replaced without it.  Since paivana
                              prefers it when both are present, its
                              for/proto/host must also reach the
                              X-Forwarded-* headers, or an origin that
                              speaks only those would be told the
                              proxy was the client.  A chain
                              containing a hop X-Forwarded-For cannot
                              express (§6.3 "unknown") yields no
                              synthesized chain rather than one with a
                              hop silently missing.
  unix socket, Forwarded      unlike X-Forwarded-For, RFC 7239 can
                              name an address-less hop, so paivana's
                              own element reads for=unknown rather
                              than being omitted.
  TRUSTED_PROXIES startup     a policy that parses to nothing usable
                              (missing trailing ';', a /0 network, an
                              address of the wrong family, junk) must
                              abort startup rather than silently
                              trusting nobody; usable ones must start.
  WHITELIST startup           an expression regcomp(3) cannot compile
                              must abort startup rather than leave
                              paivana matching against an
                              uninitialised regex_t; usable ones must
                              start.  Two of the refused cases are
                              the anchoring: "a)|(b" and "(a$|^b" do
                              not balance on their own, so wrapping
                              them in "^(%s)$" yields an alternation
                              that has climbed out of the group and a
                              whitelist matching far more than it
                              says.  paivana compiles the value bare
                              first for that reason, which is what
                              these two reach.

                              The matching itself is still out of
                              reach here: the regexec sits behind the
                              paywall that `-n` switches off, and
                              without `-n` paivana needs a merchant
                              backend to serve it templates before it
                              will start at all.  So the anchoring at
                              *match* time -- a WHITELIST of "/free/"
                              waiving payment for every URL merely
                              containing it -- has no end-to-end
                              regression test.

Cross-cutting tests (run once):

  POST /.well-known/paivana   with `-n` the payment endpoint answers
                              501 rather than falling through to the
                              proxy -- the one paywall-side branch
                              `-n` does not shield.  Both ways to get
                              it wrong are silent: forwarding the POST
                              would hand the origin payment data it
                              has no business seeing, and claiming the
                              path for every method would shadow
                              whatever the origin serves there, so the
                              GET of the same path is checked to still
                              be forwarded.

  TRACE method                unsupported HTTP verb yields 405 Method
                              Not Allowed (paivana rejects it, the
                              upstream is never contacted)
  2 MiB POST upload           request bodies above the 1 MiB
                              MAX_REQUEST_SIZE are rejected with
                              413 Content Too Large
  curl keep-alive x3          three GETs over one keep-alive TCP
                              connection all succeed
  wget /hello                 third-party client interop
  HTTP/1.1 pipelining (x4)    four requests sent back-to-back on a
                              single TCP connection *before* reading
                              any response; responses must come back
                              in the same order and with the correct
                              status codes (200, 201, 200, 404).
                              This specifically tests that paivana's
                              per-request state machine and MHD's
                              keep-alive handling cooperate correctly.
  upstream down               with paivana pointed at a closed port,
                              clients receive 502 Bad Gateway with
                              the built-in "Bad Gateway" HTML body
  early upstream response     against early_response_upstream, a
                              768 KiB POST that the upstream answers
                              with a 413 BEFORE reading the body.
                              Paivana must forward that 413 to the
                              client rather than turning it into a
                              502 — the early-response path, i.e.
                              #UP_DRAINING.
                              Note that the upstream is NOT expected
                              to see the whole body: RFC 9110 §9.3
                              lets a client stop sending once it has
                              a final response, and libcurl does
                              (plain curl against this upstream
                              sends ~128 KiB of the 768 KiB and
                              stops).  The receipt is a diagnostic;
                              the test only requires that it appear,
                              i.e. that the exchange finished
                              upstream-side.
  early response, no drain    the same, with --no-drain: having
                              answered, the upstream never reads
                              again, so paivana's outbound socket
                              stays full with a request it can no
                              longer finish sending.  It must still
                              answer its own client, promptly, with
                              the upstream's 413.  Bounded by
                              timeout(1) rather than curl --max-time
                              because the failure mode is a hang:
                              paivana's stall watchdog would
                              eventually turn it into a truncated
                              response, which must not be allowed to
                              look like a slow pass.  The drain-mode
                              case above cannot catch this — the
                              upstream there keeps reading, so the
                              socket never stays full.

The streaming tests (`test_streaming`)
--------------------------------------

Everything above would pass equally well against the fully-buffered
proxy this replaced: every body in it fits in one buffer.  These
cases are about what is new — that a body is no longer bounded by
memory, and that it starts reaching the client before the origin has
finished sending it.

  200 MiB, Content-Length     five times the 40 MiB ceiling that used
                              to make this a 502 outright.  Body
                              verified byte for byte, and the
                              origin's own Content-Length must reach
                              the client rather than one recomputed
                              from an assembled buffer.
  200 MiB, chunked            the same body without a declared
                              length; must stay chunked to the
                              client instead of being silently
                              converted.
  chunked to an HTTP/1.0      an HTTP/1.0 client cannot be sent
    client                    chunks, so the close of the connection
                              has to be the framing.
  Range -> 206                Content-Range and the partial body pass
                              through the streamed path.
  HEAD on a large resource    MHD does not run the content reader for
                              a HEAD but does emit the size the
                              response was created with, so the
                              length the equivalent GET would have
                              had now reaches the client (RFC 9110
                              §9.3.2).  Buffering could only ever
                              report 0 here.
  204 / 304                   no body either way; the 304 still
                              carries the length of the body it does
                              not send (RFC 9110 §8.6).
  200 MiB upload, both        the request body is streamed too, so
    framings                  the origin sees it byte-exact and sees
                              the client's own framing reproduced --
                              a declared length stays declared,
                              chunked stays chunked.
  small POST                  the overwhelmingly common case, which
                              now takes the same path as the large
                              one.
  chunked upstream stops      the status is long gone by the time the
    mid-stream                origin gives up, so the only remaining
                              way to say "incomplete" is to close
                              without the terminating chunk.  curl 18
                              is the client noticing.
  upstream goes quiet         MHD will not time this out (a suspended
                              connection is off its timeout lists)
                              and CURLOPT_TIMEOUT is deliberately
                              unset, so paivana's own stall watchdog
                              is the only thing that can end it.
  upstream never answers      distinct from an upstream that is not
                              there, which is a 502: this is a 504,
                              and the time-to-headers clock is what
                              tells them apart.
  100 abandoned downloads     the ownership handshake between MHD's
                              completion notifier and the content
                              reader's free callback runs on every
                              request now, so a mistake in it is a
                              use-after-free or a leak on all
                              traffic.  RSS across a hundred
                              abandoned transfers is the cheap
                              detector; ASan is the thorough one.
  abandoned upload            a Content-Length was declared upstream
                              that can no longer be delivered; the
                              origin has to be told the request is
                              broken rather than left waiting.
  early 413 during a          only reachable because the request body
    200 MiB upload            is streamed: with it buffered first the
                              origin could not have answered before
                              seeing all of it.
  trailers, 1xx               what patch 0034 established, re-checked
                              on the streamed path: neither may be
                              merged into a response that has already
                              been queued.

The congestion tests (`test_congestion`)
----------------------------------------

`test_streaming` shows that a large body gets through intact.  It does
not show that it got through *without being held in memory*, and every
case in it would pass against a version that quietly buffered the lot
-- so on their own they leave the central claim of the change untested.
These are the cases that test it.

Three things are measured that the client cannot see for itself:

  paivana's VmRSS while a body many times the buffer size is in
  flight.  This is the bound, stated directly.

  How long the *origin* took to write its body, which `stream_upstream`
  reports per connection on stderr ("served target=... bytes=N ms=M").
  A proxy that buffers takes everything at line rate however slowly its
  own client reads; one that relays can only take what the client has
  made room for.  From the client end the two are indistinguishable,
  which is why the origin has to report its own timing.

  paivana's CPU time across an interval when nothing is moving.
  Busy-waiting is the classic failure of a suspend/resume design and is
  otherwise invisible: the transfer still completes, correctly, with a
  core pinned for its duration.

Rate limits (`--read-rate`, `--upload-rate` on the client, `rate=` on
the upstream) are what make any of this reproducible.  On loopback with
both ends going flat out, the kernel socket buffers absorb everything
and no ring ever fills.

The RSS bounds are skipped under `--enable-sanitizers`.  ASan's
quarantine -- the thing that lets it catch a use-after-free -- holds
freed chunks rather than reusing them, so RSS there tracks total bytes
moved instead of bytes held: the 64 MiB case grows ~58 MB instrumented
against ~0.5 MB not, for identical code.  The transfers still run and
LSan still watches them; the pacing and CPU assertions are unaffected
and are checked in both builds.

These sizes are deliberately *not* divided by PAIVANA_TEST_SCALE.  Each
case is rate-limited, so its duration is set by the rate and not by the
size, and the sanitised build is no slower for them.  Scaling them
would also break the pacing assertions: the kernel socket buffers hold
a fixed couple of megabytes however small the body is, so at a
twentieth of the size the origin legitimately finishes well ahead of
the client and "was it throttled" stops having a stable answer.

  64 MiB through a slow      peak RSS over baseline must stay within a
    client                   few megabytes.  Measured: +552 kB across
                             20 samples, against a hard 502 for this
                             size before the change.  Checked against a
                             build with the ring cap removed, which
                             grows 12544k -> 78312k for the same body:
                             the bound does detect buffering.
  upstream pacing            the same transfer from the other end: the
                             origin's own elapsed time must track the
                             client's rather than finishing in a
                             fiftieth of it.  Measured: 3597 ms to
                             write 64 MiB to a client that read for
                             4000 ms, where buffering would have taken
                             under 100 ms on loopback.
  upload pacing              the same assertion in the request
                             direction, against /sink.
  32 concurrent throttled    the per-request cost is what multiplies,
    downloads                so this is where a bound that holds for
                             one request and not for thirty-two shows.
                             Mixed rates, so the fast ones finish while
                             the slow ones are still going.  Measured:
                             10.6 MB of growth for 32 x 8 MiB in
                             flight, about 339 kB each.  Runs with
                             PER_IP_CONNECTION_LIMIT lifted, which is
                             otherwise exactly 32 and would have the
                             case measure connection limiting instead.
  idle transfer              an origin dribbling 200 B/s leaves paivana
                             with nothing to do for ~5 s.  CPU must
                             stay near zero (measured: 1 jiffy, i.e.
                             10 ms, over 5107 ms), and the first byte
                             must still arrive at once -- measured at
                             1 ms -- rather than after the last.
  1 KiB receive buffer       makes libcurl drain paivana's socket in
                             tiny units, so MHD's content reader is
                             called hundreds of times where the default
                             buffer needs a handful -- each one a
                             chance for the ring to empty and the
                             connection to suspend and resume.  Chunked,
                             so MHD's chunk framing is re-entered every
                             time.
  slow at both ends          neither side able to keep up with the
                             other on one request.  Both rings spend
                             the transfer alternately full and empty
                             and the two halves of the state machine
                             have to interleave without deadlocking or
                             dropping a byte.

The base64url cross-check (`test_base64url.sh`)
-----------------------------------------------

The paivana ID is `<expiration>-<base64url(sha256(...))>`.  The daemon
builds it with `GNUNET_STRINGS_base64url_encode()`; the browser rebuilds
it in `paywall.js` to recognise the payment it has just made.  Nothing
in either program forces the two encoders to agree, and if they do not,
the ID never matches, the payment appears not to go through, and
neither side logs anything wrong.

They have already disagreed twice.  Once on the decode side: the daemon
emits the RFC 4648 section 5 (URL-safe) alphabet, and the browser fed it
to `atob()`, which only knows section 4 and throws on `-` or `_` -- at
module scope, so the whole script died and the paywall could not be
paid.  Once on the encode side: the browser used
`Uint8Array.prototype.toBase64`, which is a 2024-25 addition (Firefox
133, Safari 18.2, Chrome 140) and is simply not a function on anything
older.  Two bugs of the same shape in one file is what this test is for.

`base64url_vectors` prints 369 vectors as the *daemon* produces them --
every length from 0 to 96, so both amounts of padding and none are
crossed; every single byte value, because `-` and `_` are only
reachable from particular high bit patterns and are exactly the two
characters the section 4 alphabet spells differently; and sixteen
32-byte blocks, that being the size which actually occurs.
`test_base64url.sh` lifts `base64url()` out of `paywall.js` by matching
braces -- rather than keeping a copy here, which would be a second
implementation to hold in step, and holding implementations in step by
hand is the thing that failed -- and compares.  It refuses to pass on
fewer than 300 vectors, so a generator that broke would fail rather
than trivially agree.  Skips (77) without node.

Checked by breaking it: with the alphabet translation removed from
`paywall.js`, it reports `paywall.js gave "WH2ix+wRNls", the daemon
gives "WH2ix-wRNls"`.

The client_address unit test
----------------------------

`test_client_address.c` covers PAIVANA_HTTPD_resolve_forwarding(),
the single walk over the forwarding chain that decides both the client
address PAIVANA_HTTPD_get_client_address() hands to the cookie MAC and
the scheme and authority PAIVANA_HTTPD_get_base_url() rebuilds the
website string from.  That function is deliberately pure — it takes
the socket peer, the ordered field lines of each forwarding header and
the trust configuration, and nothing else — so the whole policy is
reachable without an MHD connection; the MHD half is a thin adapter.

The access cookie is an HMAC over (expiration, website, client
address).  A host therefore has to produce the *same bytes* however
paivana learns its address, or the cookie it was issued silently stops
verifying and the visitor is asked to pay again.  The test asserts:

  - an X-Forwarded-For value and the socket address of the same host
    yield identical bytes (including ::ffff:a.b.c.d from a dual-stack
    listener versus a.b.c.d from a proxy),
  - alternative spellings of one address are one identity
    ("::1" / "0:0:0:0:0:0:0:1", upper/lower case hex),
  - a value that is not a bare IP address is refused rather than
    turned into an identity of its own (port suffixes, brackets, RFC
    7239 "unknown"/"_hidden", hostnames, zone ids, junk),
  - a cookie issued for one host is not accepted for another.

A table-driven group then covers the walk itself.  `-f` means "we are
behind a trusted reverse proxy", so the socket peer is trusted
implicitly and TRUSTED_PROXIES / TRUSTED_PROXIES6 name the *additional*
hops further out; the walk steps leftwards over a node only while that
node is trusted and stops at the first one that is not.  The rows
cover:

  - without `-f`, the socket peer wins even with every forwarding
    header present,
  - a single proxy and a single element, in both spellings,
  - two and three proxies with only some of them listed, and an
    untrusted node in the middle, which stops the walk where it should,
  - a chain of nothing but trusted hops, where the leftmost is all
    there is,
  - IPv4, bracketed IPv6, IPv6 with a port, RFC 7239 §6.3 "unknown"
    and an obfuscated identifier,
  - repeated field lines of one header and a field line that is itself
    a list (RFC 9110 §5.3),
  - quoted strings with escapes, and an unterminated one, which used
    to be read past the end of the header,
  - malformed, empty and whitespace-only headers, all of which fall
    back to the socket peer rather than losing it,
  - `Forwarded` winning where both headers are present,
  - a chain of 2500 elements, which is refused outright rather than
    walked: every element used to be located by rescanning the header
    from byte 0.

A second table covers what the base URL is built from: `proto=` and
`host=` taken from the same element the address came from, the
X-Forwarded-Proto/-Host/-Port fallbacks, an X-Forwarded-Host that
already carries a port together with an X-Forwarded-Port (which must
not yield "example.com:8443:8443"), ports re-rendered rather than
echoed, and hosts and schemes that are refused because they are not
one.

A further group covers the rendering back out — a `for=` identifier,
an X-Forwarded-For chain, and RFC 7239 §4 values, where a parameter
that would otherwise splice a second forwarded-element into a header
we build is either quoted or reported as absent.

A separate group pins the behaviour of GNUnet's
GNUNET_STRINGS_parse_ipv{4,6}_policy() that load_trusted_proxies()
compensates for: the mandatory trailing ';', the v4/v6 disagreement
about spaces, the two ways those parsers return "nothing usable"
without returning NULL (a /0 network, which is indistinguishable from
the list terminator, and an address of the wrong family), and the one
way they return "usable, but not what was written" — a final entry
without its ';', or anything after the last ';', is dropped and the
prefix reported as success, which is why the loader counts separators.
If upstream ever fixes these, this group is what says so.

The startup validation built on top of that is in the integration
suite instead, since it is about whether the daemon comes up.

It links paivana-httpd_helper.c and paivana-httpd_cookie.c directly
and supplies the daemon globals itself, so it needs no MHD connection
and no merchant backend.  The integration suite cannot cover any of
this: with `-n` the cookie path is never reached, so the client
address is never computed.

The cookie unit tests
---------------------

The same applies to the two cookie tests, and for the same reason:
`-n` sets do_forward before the request is looked at, so nothing in
the integration suite ever mints or checks a cookie.  Both link only
paivana-httpd_cookie.c.

`test_cookie_header.c` is about the header paivana emits, i.e. about
whether the credential the client just paid for ever comes back: the
`Path` re-encoding (the browser matches against the encoded request
path, while the URL paivana holds has been decoded by MHD), the RFC
6265 §4.1.1 grammar the attribute has to satisfy, attribute injection
through a path containing ';', `Secure`, and the `Max-Age` floor that
keeps a sub-second access from being deleted on arrival.

`test_cookie_access.c` is about the decision made when it does come
back.  The cookie is a bearer token we hand to the party most
interested in widening it, so each of the three things it is minted
for -- expiration, website, client address -- is checked to be inside
the MAC and re-checked on presentation, including the obvious attempt:
reading the expiration off the value and writing a later one.  Each of
the ways check_cookie() can reject a value has its own case, so that a
malformed value ends in a refusal rather than in a read past the end
of a string the client chose.  A separate group covers the values that
are not malformed at all but merely respelled -- a leading '+', a
leading space, a leading zero, junk between the seconds and the '-' --
each of which decodes to the same seconds and the same hash as a
cookie we really issued, and so is a second live spelling of one
credential unless the parser refuses it.  `-g` is covered here and
nowhere else.

The `paivana_id` is pinned against a golden vector computed
independently from the definition the paywall page implements
(src/frontend/paywall.js, makePaivanaId()).  Neither side ever sends
it; both derive it from their own copy of (nonce, website, expiration)
and expect the other to have got the same string, so the two
implementations agreeing IS the protocol, and a change on either side
that this vector does not survive means every order is created under
an id the other side will not look for.


The benchmark (`benchmark.sh`)
------------------------------

Not a test -- it asserts nothing about correctness and its result
depends on the machine.  It answers two questions: how much
throughput does putting paivana in front of an origin cost, and how
fast is paivana on its own when the paywall turns a client away?

    meson test --benchmark proxy_overhead -C build   # the first
    meson test --benchmark paywall_page   -C build   # the second

Both are registered with meson's `benchmark()` rather than `test()`,
which is what keeps them out of `make check`: `meson test` does not
run benchmarks.  They are two entries rather than one so that they
skip independently -- the paywall arm needs things the proxy arm does
not, and should not be able to take it down with it.  Run the script
directly for the knobs -- `-c` clients, `-s` page size, `-d` seconds,
`-m direct|proxy|paywall|both|all`.  It exits 77 when rustc was not
available to build upstream_rs, or when the build is sanitized (those
timings measure ASan), and 1 if any request failed, since a run with
failures has not measured throughput.

N curl workers fetch a fixed-size page for a fixed time.  In `direct`
they fetch it straight from upstream_rs, in `proxy` through paivana
in front of that same upstream_rs, and in `paywall` they fetch
paivana's own 402 page with no upstream in the path at all.  For each
arm it reports requests, requests/s, MB/s (10^6), the page size and
the server processes' CPU time; then the proxy/direct ratio, and the
paywall/proxy one.  Nothing touches disk: the upstream's page comes
from a buffer it fills once at startup -- it used to regenerate it a
byte at a time per request, which was real work charged to the arm
that has no proxy in it -- the paywall page comes out of paivana's
response cache, and the clients discard every body.

The expected size is measured from a warm-up request rather than
assumed, because in paywall mode nothing here knows it up front: it
is whatever `paywall.en.must` renders to (50300 bytes as of writing).
That measured size is then what every later response is checked
against, so a short body counts as a failure rather than as
throughput that was not achieved.

Three properties of the setup shape the number, and all three make
the proxy look better than it is, so the reported ratio is a floor:

  - Every request gets a fresh TCP connection in *both* arms.  That
    is forced, not chosen: upstream_rs answers one request per
    connection and closes, so the direct arm cannot keep-alive at
    all, while paivana's client side happily would -- MHD strips the
    upstream's hop-by-hop `Connection: close` and decides the client
    connection's fate itself.  Measured: without the clients sending
    `Connection: close`, curl's second request through paivana
    reports num_connects=0 and the same request direct reports 1.
    So the clients send it, and both arms are charged one TCP setup
    per request.

  - curl's own process startup (12.5 ms on the machine this was
    written on -- it links openssl, nghttp2, brotli, zstd, ldap) is
    amortised by handing each curl invocation a batch of URLs.  The
    batch size converges at run time rather than being computed from
    the page size, because how long a batch takes depends on the
    per-worker request rate, which is what is being measured: a size
    picked up front overshot a 3 s run by 7% at `-c 32`.  At a fixed
    batch of 8 the startup was 44% of the run and the reported rate
    came out at half the truth.

    Both of the above are per-request constants added to *both*
    arms, so they pull the ratio toward 1.

  - paivana is single-threaded by construction -- one GNUnet
    scheduler driving MHD and libcurl -- and upstream_rs spawns a
    thread per connection, so on a multi-core box the direct arm may
    use every core and the proxy arm may not.  That is a real
    property of paivana rather than an artefact of the harness, but
    it does make the ratio a function of the core count, which is
    why CPU seconds and `nproc` are printed with it.

A fourth applies to paywall mode only: the paywall page is not the
`-s` size, so that arm is comparable to the others in requests/s and
not in MB/s.  The script prints both page sizes next to the ratio and
does not offer a MB/s one.

For orientation, one run on a 24-core machine at the defaults (8
clients, 64 KiB page, 10 s):

    direct    34932 req/s   2289 MB/s   upstream_rs   3.07 cores
    proxy      6816 req/s    447 MB/s   paivana       0.97 cores
    paywall   33993 req/s   1710 MB/s   paivana       0.90 cores

so 0.20x of direct through the proxy, and 4.99x the forwarded rate
for the paywall page (50300 bytes).  paivana is at 0.97 cores
forwarding: it is saturating its single thread, which is the bound
that matters.  Sweeping the paywall arm shows the same bound from the
other side -- 26852 req/s at `-c 4` and 0.81 cores, 33504 at `-c 8`,
36049 at `-c 16` and 0.98 cores, 35892 at `-c 32` -- i.e. it stops
scaling exactly where the thread runs out, at about 36k req/s.

Do not quote those figures.  They are stable to a couple of percent
when the machine is quiet, but an *earlier* set on the same 24-core
box read 13375 req/s direct and 4707 through paivana, i.e. 0.35x
rather than 0.20x, because the direct arm was then getting only 1.5
cores instead of 3.1.  The clients are a bash loop forking curl and
they compete with the servers for the machine, so under contention
the fastest arm loses the most and the ratio flatters the proxy.  The
run's own CPU numbers are what tell you which regime you were in.

Why the paywall arm needs a merchant backend, and why a stub is
honest here.  paivana does not open its listen socket until it has
fetched a template from a merchant backend
(PAIVANA_HTTPD_load_templates -> templates_ready ->
PAIVANA_HTTPD_serve_requests), so paywall mode cannot measure
anything without one.  It starts `merchant_stub`, which answers the
two GETs of that startup exchange -- shaped as
merchant_api_get-private-templates{,-TEMPLATE_ID}.c parse them, with
the contract test_paywall.sh POSTs to a real backend -- and nothing
else.  That is the whole of what a real backend would do here: the
template is fetched once, the page is rendered locally from it, and
the rendered MHD_Response is cached per (language, encoding) in
load_paywall(), so from the first measured request onwards a live
merchant is exactly as idle as the stub.  This benchmark never buys
anything, so no code path that can tell the two apart is reached.
The stub does check the bearer token, since paivana building that
header out of MERCHANT_ACCESS_TOKEN is the one part of the exchange
that could silently regress; a 401 makes paivana refuse to start
rather than start against a configuration nobody would deploy.  To
check the claim instead of taking it, set PAIVANA_BENCH_MERCHANT_URL
(and PAIVANA_BENCH_MERCHANT_TOKEN) at a live backend carrying a
`paivana` template -- named `premium`, or whatever
PAIVANA_BENCH_TEMPLATE_ID says.

Like test_paywall.sh, paywall mode stages `paywall.en.must` into a
scratch prefix and points PAIVANA_PREFIX at it, rather than requiring
`make install` for a page that lives in the build tree.

The likeliest cause of failures here is not paivana: one connection
per request against a fixed server port pins the 4-tuple for the
TIME_WAIT duration, and a few runs back to back can fill the local
ephemeral range (28k ports by default against ~13k connections per
run).  The script says so when it sees transfers that never
connected.


Environment variables
---------------------

The driver script honors:

  PAIVANA_HTTPD   path to paivana-httpd (default: the in-tree build)
  SRCDIR          directory containing the upstream sources and the
                  conf template (default: dirname of the script)
  BUILDDIR        directory containing upstream_mhd, pipeline_client,
                  upstream_go, upstream_rs (default: $PWD)
  KEEP_TMP=1      do not delete the scratch dir on exit
  PAIVANA_PORT_BASE
                  first port of the block the suite binds
                  (default 18400); see below

benchmark.sh honors the first four of those, plus:

  PAIVANA_BENCH_PORT_BASE
                  first port of its own block (default 18600)
  PAIVANA_BENCH_MERCHANT_URL
                  a live merchant backend for paywall mode, instead
                  of starting merchant_stub
  PAIVANA_BENCH_MERCHANT_TOKEN
                  bearer token for it
  PAIVANA_BENCH_TEMPLATE_ID
                  template to ask that backend for (default
                  `premium`, which is what merchant_stub serves)

Ports used
----------

Every port is an offset off PAIVANA_PORT_BASE, which defaults to 18400
-- the 184xx / 185xx range, chosen to avoid collisions with real
services.  The suite checks all ten before it starts anything and
exits 77 (meson reads that as SKIP) if one of them is taken, naming
it.

That check is not a formality.  Readiness used to be "does something
accept on this port", which is a different question from "did our
child come up": a paivana that lost the bind to a squatter -- most
often a stale one of its own from an earlier run -- read as started,
and the suite then ran its checks against the wrong process.  With a
stale paivana of a different vintage they even pass.  The startup
validation cases are the worst affected, since those decide "refused"
from exactly that probe.

  base + 1   (18401)   upstream_mhd
  base + 2   (18402)   upstream_go
  base + 3   (18403)   upstream_py
  base + 4   (18404)   upstream_rs
  base + 5   (18405)   early_response_upstream
  base + 6   (18406)   early_response_upstream --no-drain
  base + 7   (18407)   truncating upstream (short-body test)
  base + 8   (18408)   stream_upstream (streaming tests)
  base + 99  (18499)   dead port (for "upstream down" test)
  base + 100 (18500)   paivana-httpd

Move the base to run the suite in two checkouts at once, or beside a
paivana you are debugging:

    PAIVANA_PORT_BASE=18700 meson test -C build reverse_proxy

benchmark.sh binds its own three ports off a separate base, so it can
run beside the suite: PAIVANA_BENCH_PORT_BASE, default 18600, giving
18601 for upstream_rs, 18602 for paivana and 18603 for merchant_stub.
It checks the ones the chosen `-m` actually needs, and skips the same
way.  The `paywall_page` benchmark entry passes `-b 18610` so that
the two entries cannot collide if anyone runs the benchmarks in
parallel, which is not meson's default but is one flag away.

Endpoints (implemented by every upstream)
-----------------------------------------

  GET /hello                  text "Hello from <name>\n"
  GET /status/NNN             respond with status NNN and a trivial
                              text body "status NNN\n"
  GET /large/N                N bytes of 'A'..'Z' repeating
  GET /slow/N                 sleep N ms, then "slept\n"
  GET /echo-headers           text listing of received request
                              headers, "Key: Value\n" per line
  POST /echo                  body is echoed verbatim
  POST /upload                "Received N bytes\n"
  PUT /put                    "PUT received N\n"
  PATCH /patch                "PATCH received N\n"
  DELETE /item*               204 No Content
  OPTIONS *                   204 + Allow: GET, POST, PUT, ...

Every response also carries an `X-Upstream:` header whose value
identifies which server handled it (mhd, go, py, rs); the client
test cases use it to confirm that responses are coming back from
the expected backend.


The paywall suite
-----------------

`test_paywall.sh` covers what `-n` hides.  It puts a real GNU Taler
system behind paivana-httpd -- a fakebank, an exchange and a merchant
backend, started with `taler-unified-setup.sh` exactly as the merchant
and anastasis suites start theirs -- creates a Paivana template on the
merchant instance, buys access with `taler-wallet-cli`, and checks what
the daemon does with the result.  31 checks, about 25 seconds.

It skips (exit 77) rather than failing when the environment cannot
support it: no `taler-unified-setup.sh`, `taler-wallet-cli`,
`taler-merchant-httpd`, `jq`, `python3` or PostgreSQL, no built paywall
template, or one of its ports already in use.  A skip names what was
missing.

Ports.  paivana's own two move with `PAIVANA_PORT_BASE` (+110 and
+111), but the Taler system's are fixed at 9966 (merchant), 8081
(exchange) and 8082 (bank) -- the same ones the merchant suite uses, so
the two cannot run at once and this suite skips when they are busy.  It
also wants a PostgreSQL database named `paivanacheck`, which it creates
if it can; `talercheck` is deliberately not reused, since the merchant's
own tests would then be clobbering these tables and vice versa.

The paywall template is staged into a throwaway prefix and reached
through `PAIVANA_PREFIX`, so a build tree is enough and `make install`
is not required.

Why the client half is written out by hand.  The paywall page computes
a payment identifier from (nonce, website, expiration) and the daemon
computes the same identifier independently; neither ever sends it to
the other, so the two agreeing IS the protocol.  `paivana_id.py`
re-derives it -- and the Crockford base32 encoding of the nonce -- from
the definition `src/frontend/paywall.js` implements, which is what makes
this a test of both ends rather than of one end twice.  It agrees with
the golden vector in `test_cookie_access.c`, which was computed the same
way; if you change the derivation, three places have to move together.

What it covers, in order:

  * the unpaid path: 302 to the paywall, the template named in the
    Location and the website base64url-encoded in the fragment, the
    402 page itself, its `Paivana:` pay-template URI and its CSP;
  * the whitelist, and specifically that a WHITELIST expression is
    anchored at both ends -- `/echo-headers` waives that path and not
    `/x/echo-headers` or `/echo-headers/x`.  The regexec that decides
    this sits behind the paywall, so no other test in the tree can
    reach it;
  * the redemption endpoint's refusals: a body missing its fields, a
    nonce of the wrong length, an order the merchant never saw;
  * a real payment, redeemed for a real access cookie, and that cookie
    opening the URL it was minted for and no other;
  * that rewriting the expiration in the cookie value invalidates it
    (the expiration is the KDF salt) and that a malformed cookie is
    refused rather than mis-parsed;
  * an order bought for a DIFFERENT fulfillment URL under the session
    we then claim.  The merchant sells it, the session lookup succeeds,
    and the only thing between that and a cookie for a page nobody paid
    for is paivana comparing the contract's fulfillment URL against the
    website claimed.  This is the one case that reaches that comparison:
    naming another website in the redemption changes the payment
    identifier, so every simpler attempt is refused earlier, by the
    session lookup;
  * that redemption is repeatable from anywhere, which is deliberate
    (design document 076, "Payment buys access, not a seat").  The
    check is here so that a change of mind about it surfaces as a test
    failure rather than as a silent change of policy.

The checks were verified not to be vacuous by breaking the code under
them, one property at a time: dropping the `^(...)$` wrapping around
WHITELIST turns the two anchoring cases red (`/x/echo-headers` reaches
the origin); dropping the website from the cookie's keyed hash lets the
paid cookie open `/item` as well; and skipping the fulfillment-URL
comparison lets an order bought for `/elsewhere` mint a cookie for
`/item`.
