.. mode: -*- rst -*-

Telemetry
=========

:Tag: design.mps.telemetry
:Author: Gavin Matthews
:Date: 1997-04-11
:Status: incomplete design
:Revision: $Id$
:Copyright: See `Copyright and License`_.
:Index terms:   pair: telemetry; design


Introduction
------------

_`.intro`: This documents the design of the telemetry mechanism within
the MPS.

_`.readership`: This document is intended for any MPS developer.

_`.source`: Various meetings and brainstorms, including
meeting.general.1997-03-04(0), `mail.richard.1997-07-03.17-01`_,
`mail.gavinm.1997-05-01.12-40`_.

.. _mail.gavinm.1997-05-01.12-40: https://info.ravenbrook.com/project/mps/mail/1997/05/01/12-40/0.txt
.. _mail.richard.1997-07-03.17-01: https://info.ravenbrook.com/project/mps/mail/1997/07/03/17-01/0.txt


Overview
--------

_`.over`: Telemetry permits the emission of events from the MPS. These
can be used to drive a graphical tool, or to debug, or whatever. The
system is flexible and robust, but doesn't require heavy support from
the client.


Requirements
------------

_`.req.simple`: It must be possible to generate code both for the MPS
and any tool without using complicated build tools.

_`.req.open`: We must not constrain the nature of events before we are
certain of what we want them to be.

_`.req.multi`: We must be able to send events to multiple streams.

_`.req.share`: It must be possible to share event descriptions between
the MPS and any tool.

_`.req.version`: It must be possible to version the set of events so
that any tool can detect whether it can understand the MPS.

_`.req.back`: Tools should be able to understand older and newer
version of the MPS, so far as is appropriate.

_`.req.type`: It must be possible to transmit a rich variety of types
to the tool, including doubles, and strings.

_`.req.port`: It must be possible to transmit and receive events
between different platforms.

_`.req.control`: It must be possible to control whether and what
events are transmitted at least at a coarse level.

_`.req.examine`: There should be a cheap means to examine the contents
of logs.

_`.req.pm`: The event mechanism should provide for post mortem to
detect what significant events led up to death.

_`.req.perf`: Events should not have a significant effect on
performance when unwanted.

_`.req.small`: Telemetry streams should be small.

_`.req.avail`: Events should be available in all varieties, subject to
performance requirements.

_`.req.impl`: The plinth support for telemetry should be easy to write
and flexible.

_`.req.robust`: The telemetry protocol should be robust against some
forms of corruption, e.g. packet loss.

_`.req.intern`: It should be possible to support string-interning.


Architecture
------------

_`.arch`: Event annotations are scattered throughout the code, but
there is a central registration of event types and properties. Events
are written to a buffer via a specialist structure, and are optionally
written to the plinth. Events can take any number of parameters of a
range of types, indicated as a format both in the annotation and the
registry.


Analysis
--------

_`.analysis`: The proposed order of development, with summary of
requirements impact is as follows (★ for positive impact, ⇓ for
negative impact):

========================= == == == == == == == == == == == == == == == == == =======
solution                  si op mu sh ve ty po co ex pm pe sm av im ro in ba status
========================= == == == == == == == == == == == == == == == == == =======
`.sol.format`_            ·  ·  ·  ·  ·  ★  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  merged
`.sol.struct`_            ·  ·  ·  ·  ·  ★  ·  ·  ·  ·  ★  ⇓  ·  ·  ·  ·  ·  merged
`.sol.string`_            ·  ·  ·  ·  ·  ★  ·  ·  ·  ·  ·  ·  ·  ·  ·  ★  ·  merged
`.sol.relation`_          ★  ·  ·  ★  ·  ·  ·  ·  ★  ·  ·  ★  ·  ·  ·  ·  ·  merged
`.sol.dumper`_            ·  ·  ·  ·  ·  ·  ·  ·  ★  ·  ·  ·  ·  ·  ·  ·  ·  merged
`.sol.kind`_              ·  ⇓  ·  ·  ·  ·  ·  ★  ·  ★  ·  ·  ·  ·  ·  ·  ·  merged
`.sol.control`_           ·  ·  ·  ·  ·  ·  ·  ★  ·  ·  ★  ·  ·  ·  ·  ·  ·  merged
`.sol.variety`_           ·  ·  ·  ·  ·  ·  ·  ·  ·  ★  ★  ·  ★  ·  ·  ·  ·
========================= == == == == == == == == == == == == == == == == == =======

The following are not yet ordered:

========================= == == == == == == == == == == == == == == == == == =======
solution                  si op mu sh ve ty po co ex pm pe sm av im ro in ba status
========================= == == == == == == == == == == == == == == == == == =======
`.sol.buffer`_            ·  ·  ·  ·  ·  ·  ·  ★  ·  ★  ★  ·  ·  ·  ·  ·  .
`.sol.traceback`_         ·  ·  ·  ·  ·  ·  ·  ·  ·  ★  ·  ·  ·  ·  ·  ·  .
`.sol.client`_            ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ★  .
`.sol.head`_              ·  ·  ·  ·  ·  ·  ★  ·  ·  ·  ·  ·  ·  ·  ·  ·  .
`.sol.version`_           ·  ·  ·  ·  ★  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ★
`.sol.exit`_              ·  ·  ·  ·  ·  ·  ·  ·  ·  ★  ·  ·  ·  ·  ·  ·  .
`.sol.block`_             ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ★  ⇓  ·  ·  ★  ·  ·
`.sol.code`_              ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ·  ★  ·  ·  ·  ·  ★
`.sol.msg`_               ·  ·  ★  ·  ·  ·  ★  ·  ·  ·  ·  ·  ·  ★  ★  ·  .
========================= == == == == == == == == == == == == == == == == == =======

_`.file-format`: One of the objectives of this plan is to minimise the
impact of the changes to the log file format. This is to be achieved
firstly by completing all necessary support before changes are
initiated, and secondly by performing all changes at the same time.


Ideas
-----

_`.sol.format`: Event annotations indicate the types of their
arguments, for example, ``EVENT_WD`` for a ``Word`` and a ``double``.
(`.req.type`_)

_`.sol.struct`: Copy event data into a structure of the appropriate
type, for example, ``EventWDStruct``. (`.req.type`_, `.req.perf`_, but
not `.req.small`_ because of padding)

_`.sol.string`: Permit at most one string per event, at the end, and
use the ``char[1]`` hack, and specialised code; deduce the string
length from the event length and also ``NUL``-terminate (`.req.type`_,
`.req.intern`_)

_`.sol.buffer`: Enter all events initially into internal buffers, and
conditionally send them to the message stream. (`.req.pm`_,
`.req.control`_, `.req.perf`_)

_`.sol.variety`: In optimized varieties, have internal events (see
`.sol.buffer`_) for a subset of events and no external events; in
normal varieties have all internal events, and the potential for
external events. (`.req.avail`_, `.req.pm`_, `.req.perf`_)

_`.sol.kind`: Divide events by some coarse type into around 6 groups,
probably related to frequency. (`.req.control`_, `.req.pm`_, but not
`.req.open`_)

_`.sol.control`: Hold flags to determine which events are emitted
externally. (`.req.control`_, `.req.perf`_)

_`.sol.dumper`: Write a simple tool to dump event logs as text.
(`.req.examine`_)

_`.sol.msg`: Redesign the plinth interface to send and receive
messages, based on any underlying IPC mechanism, for example, append
to file, TCP/IP, messages, shared memory. (`.req.robust`_,
`.req.impl`_, `.req.port`_, `.req.multi`_)

_`.sol.block`: Buffer the events and send them as fixed size blocks,
commencing with a timestamp, and ending with padding. (`.req.robust`_,
`.req.perf`_, but not `.req.small`_)

_`.sol.code`: Commence each event with two bytes of event code, and
two bytes of length. (`.req.small`_, `.req.back`_)

_`.sol.head`: Commence each event stream with a platform-independent
header block giving information about the session, version (see
`.sol.version`_), and file format; file format will be sufficient to
decode the (platform-dependent) rest of the file. (`.req.port`_)

_`.sol.exit`: Provide a mechanism to flush events in the event of
graceful sudden death. (`.req.pm`_)

_`.sol.version`: Maintain a three part version number for the file
comprising major (incremented when the format of the entire file
changes (other than platform differences)), median (incremented when
an existing event changes its form or semantics), and minor
(incremented when a new event type is added); tools should normally
fail when the median or major is unsupported. (`.req.version`_,
`.req.back`_)

_`.sol.relation`: Event types will be defined in terms of a relation
specifying their name, code, optimised behaviour (see
`.sol.variety`_), kind (see `.sol.kind`_), and format (see
`.sol.format`_); both the MPS and tool can use this by suitable
``#define`` hacks. (`.req.simple`_, `.req.share`_, `.req.examine`_,
`.req.small`_ (no format information in messages))

_`.sol.traceback`: Provide a mechanism to output recent events (see
`.sol.buffer`_) as a form of backtrace when ``AVER`` statements fire
or from a debugger, or whatever. (`.req.pm`_)

_`.sol.client`: Provide a mechanism for user events. (`.req.intern`_)



Implementation
--------------

Annotation
..........

_`.annot`: An event annotation is of the form::

    EVENT3(FooCreate, pointer, address, word)

_`.annot.string`: If there is a string in the format, it must be the
last parameter (and hence there can be only one). There is currently
a maximum string length, defined by ``EventMaxStringLength`` in
impl.h.eventcom.

_`.annot.type`: The event type should be given as the first parameter
to the event macro, as registered in impl.h.eventdef.

_`.annot.param`: The parameters of the event should be given as the
remaining parameters of the event macro, in order as indicated in the
event parameters definition in impl.h.eventdef.


Registration
............

_`.reg`: All event types and parameters should be registered in
impl.h.eventdef, in the form of a higher-order list macros.

_`.reg.just`: This use of a higher-order macros enables great
flexibility in the use of this file.

_`.reg.rel`: The event type registration is of the form::

    EVENT(X, FooCreate, 0x1234, TRUE, Arena)

_`.reg.type`: The first parameter of the relation is the event type.
This needs no prefix, and should correspond to that used in the
annotation.

_`.reg.code`: The second parameter is the event code, a 16-bit value
used to represent this event type. Codes should not be re-used for new
event types, to allow interpretation of event log files of all ages.

_`.reg.always`: The third parameter is a boolean value indicating
whether this event type should be implemented in all varieties. See
`.control.buffer`_. Unless your event is on the critical path
(typically per reference or per object), you will want this to be
``TRUE``.

_`.reg.kind`: The fourth parameter is a kind keyword indicating what
category this event falls into. See `.control`_. The possible values
are:

- ``Arena`` -- per space or arena or global
- ``Pool`` -- pool-related
- ``Trace`` -- per trace or scan
- ``Seg`` -- per segment
- ``Ref`` -- per reference or fix
- ``Object`` -- per object or allocation
- ``User`` -- invoked by the user through the MPS interface

This list can be seen in impl.h.eventcom.

_`.reg.doc`: Add a docstring column. [RB 2012-09-03]

_`.reg.params`: The event parameters registration is of the form::

    #define EVENT_FooCreate_PARAMS(PARAM, X) \
      PARAM(X,  0, P, firstParamPointer) \
      PARAM(X,  1, U, secondParamUnsigned)

_`.reg.param.index`: The first column is the index, and must start at
zero and increase by one for each row.

_`.reg.param.sort`: The second column is the parameter "sort", which,
when appended to ``EventF``, yields a type for the parameter. It is a
letter from the following list:

- ``P`` -- ``void *``
- ``A`` -- ``Addr``
- ``W`` -- ``Word``
- ``U`` -- ``unsigned int``
- ``S`` -- ``char *``
- ``D`` -- ``double``
- ``B`` -- ``Bool``

The corresponding event parameter must be assignment compatible with
the type.

_`.param.types`: When an event has parameters whose type is not in the
above list, use the following guidelines: All ``C`` pointer types not
representing strings use ``P``; ``Size``, ``Count``, ``Index`` use
``W``; others should be obvious.

_`.reg.param.name`: The third column is the parameter name. It should
be a valid C identifier and is used for debugging display and human
readable output.

_`.reg.param.doc`: Add a docstring column. [RB 2012-09-03]

_`.reg.dup`: It is permissible for the one event type to be used for
more than one annotation. There are generally two reasons for this:

- Variable control flow for successful function completion;
- Platform/Otherwise-dependent implementations of a function.

Note that all annotations for one event type must have the same format
(as implied by `.sol.format`_).


Control
.......

_`.control`: There are two types of event control, buffer and output.

_`.control.buffer`: Buffer control affects whether particular events
implemented at all, and is controlled statically by variety using the
always value (see `.reg.always`_) for the event type. The hot variety
does compiles out annotations with ``always=FALSE``. The cool variety
does not, so always buffers a complete set of events.

_`.control.output`: Output control affects whether events written to
the internal buffer are output via the plinth. This is set on a
per-kind basis (see `.reg.kind`_), using a control bit table stored in
EventKindControl. By default, all event kinds are off. You may switch
some kinds on using a debugger.

For example, to enable ``Pool`` events using gdb (see impl.h.eventcom for
numeric codes)::

    $ gdb ./xci3gc/cool/amcss
    (gdb) break GlobalsInit
    (gdb) run
    ...
    (gdb) print EventKindControl |= 2
    $2 = 2
    (gdb) continue
    ...
    (gdb) quit
    $ mpseventcnv -v | sort | head
    0000178EA03ACF6D PoolInit                9C1E0    9C000 0005E040
    0000178EA03C2825 PoolInitMFS             9C0D8    9C000     1000        C
    0000178EA03C2C27 PoolInitMFS             9C14C    9C000     1000       44
    0000178EA03C332C PoolInitMV              9C080    9C000     1000       20    10000
    0000178EA03F4DB4 BufferInit             2FE2C4   2FE1B0        0
    0000178EA03F4EC8 BufferInitSeg          2FE2C4   2FE1B0        0
    0000178EA03F57DA AMCGenCreate           2FE1B0   2FE288
    0000178EA03F67B5 BufferInit             2FE374   2FE1B0        0
    0000178EA03F6827 BufferInitSeg          2FE374   2FE1B0        0
    0000178EA03F6B72 AMCGenCreate           2FE1B0   2FE338

_`.control.env`: The initial value of ``EventKindControl`` is read
from the C environment when the ANSI Plinth is used, and so event
output can be controlled like this::

    MPS_TELEMETRY_CONTROL=127 amcss

or like this::

    MPS_TELEMETRY_CONTROL="Pool Arena" amcss

where the variable is set to a space-separated list of names defined by ``EventKindENUM``.

_`.control.just`: These controls are coarse, but very cheap.

_`.control.external`: The MPS interface functions
``mps_telemetry_set()`` and ``mps_telemetry_reset()`` can be used to
change ``EventKindControl``.

_`.control.tool`: The tools will be able to control
``EventKindControl``.


Debugging
.........

_`.debug.buffer`: Each event kind is logged in a separate buffer,
``EventBuffer[kind]``.

_`.debug.buffer.reverse`: The events are logged in reverse order from
the top of the buffer, with the last logged event at
``EventLast[kind]``. This allows recovery of the list of recent events
using the ``event->any.size`` field.

_`.debug.dump`: The contents of all buffers can be dumped with the
``EventDump`` function from a debugger, for example::

    gdb> print EventDump(mps_lib_get_stdout())

_`.debug.describe`: Individual events can be described with the
EventDescribe function, for example::

    gdb> print EventDescribe(EventLast[3], mps_lib_get_stdout(), 0)

_`.debug.core`: The event buffers are preserved in core dumps and can
be used to work out what the MPS was doing before a crash. Since the
kinds correspond to frequencies, ancient events may still be available
in some buffers, even if they have been flushed to the output stream.
Some digging may be required.


Dumper tool
...........

_`.dumper`: A primitive dumper tool is available in impl.c.eventcnv.
For details, see guide.mps.telemetry.


Allocation replayer tool
........................

_`.replayer`: A tool for replaying an allocation sequence from a log
is available in impl.c.replay.


Document History
----------------

- 1997-04-11 Gavin Matthews. Incomplete design.

- 1997-07-07 Gavin Matthews. Rewritten after discussion in Pool Hall.

- 2002-06-07 RB_ Converted from MMInfo database design document.

- 2012-09-03 RB_ Removed basic untruths and added some discussion of
  debugging, though this starts to resemble a manual rather than a
  design document, and needs to be reworked.

- 2013-05-22 GDR_ Converted to reStructuredText.

.. _RB: https://www.ravenbrook.com/consultants/rb/
.. _GDR: https://www.ravenbrook.com/consultants/gdr/


Copyright and License
---------------------

Copyright © 2013–2020 `Ravenbrook Limited <https://www.ravenbrook.com/>`_.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
