..	-*- mode: rst-mode -*-
..
.. Note: This file includes further autogenerated ones.
..
.. Version number is filled in automatically.
.. |version| replace:: 1.4

==========
BroControl
==========

.. rst-class:: opening

    This document summarizes installation and use of *BroControl*,
    Bro's interactive shell for operating Bro installations. *BroControl*
    has two modes of operation: a *stand-alone* mode for
    managing a traditional, single-system Bro setup; and a *cluster*
    mode for maintaining a multi-system setup of coordinated Bro
    instances load-balancing the work across a set of independent
    machines.  Once installed, the operation is pretty similar
    for both types; just keep in mind that if this document refers to
    "nodes" and you're in a stand-alone setup, there is only a
    single one and no worker/proxies.

.. contents::

Download
--------

You can find the latest BroControl release for download at
http://www.bro.org/download.

BroControl's git repository is located at
`git://git.bro.org/broctl <git://git.bro.org/broctl>`_. You
can browse the repository `here <http://git.bro.org/broctl.git>`_.

This document describes BroControl |version|. See the ``CHANGES``
file for version history.

Prerequisites
-------------

Running BroControl requires the following prerequisites:

  - A Unix system. FreeBSD, Linux, and Mac OS X are supported and
    should work out of the box. Other Unix systems will quite likely
    require some tweaking. Note that in a cluster setup, all systems
    must be running exactly the *same* operating system.

  - A version of *Python* >= 2.6 (on FreeBSD, the package "py27-sqlite3" must
    also be installed).

  - A *bash* (note in particular, that on FreeBSD, *bash* is not
    installed by default).

  - In a cluster setup, *rsync* must be installed on every host
    in the cluster.

  - In a cluster setup, *sshd* must be installed and running on every host
    except the manager, and *ssh* must be installed on the manager.

  - If *sendmail* is installed (for a cluster setup, it is needed on the
    manager only), then BroControl can send mail.  Otherwise, BroControl
    will not attempt to send mail.

If you're using a load-balancing method such as PF_RING, then there is
additional software to install (for details, see the
:doc:`Cluster Configuration <../../configuration/index>` documentation).

Additionally, in a cluster setup you must have the same user account set up
on all hosts, and ``ssh`` access from the manager node to this
user account must be setup on all machines, and must work
without asking for a password/passphrase (for example, using ssh
public key authentication).  Finally, on the worker nodes, this user
must have access to the target network interface in promiscuous mode.


Installation
------------

Follow the directions to install Bro and BroControl according to
the instructions in the :doc:`Installing Bro <../../install/install>`
documentation.  Depending on how you install Bro and BroControl, the files
and directories that get installed might be owned by the "root" user.
If you want to run BroControl as an ordinary user, then you will need
to make sure that the user has write permissions to the "logs" and "spool"
directories (and all subdirectories of these directories).

Note that if you are planning to run Bro in a cluster
configuration, then you need to install Bro and BroControl only on the
manager host (the BroControl install_ or deploy_ commands will install Bro
and all required scripts to the other hosts in your cluster).

For more details on how to configure BroControl in a cluster configuration,
see the examples in the
:doc:`Cluster Configuration <../../configuration/index>` documentation.


Getting Started
---------------

BroControl is an interactive interface for managing a Bro installation
which allows you to, e.g., start/stop the monitoring or update its
configuration.

Before actually running BroControl, you first need to edit the ``broctl.cfg``,
``node.cfg``, and ``networks.cfg`` files.  In the ``broctl.cfg`` file, you
should review the BroControl options and make sure the options are set
correctly for your environment (for a description of every BroControl option,
see the `Option Reference`_ section below).  Next, edit the ``node.cfg`` file
and specify the nodes that you will be running.  For a description of
every option available for nodes, see the `Node`_ section below.
Finally, edit the ``networks.cfg`` file and list each network (see
the examples in the file for the format to use) that is considered
local to the monitored environment.

BroControl is started with the ``broctl`` script and then expects
commands on its command-line (alternatively, ``broctl`` can also be started
with a single command directly on the shell's command line, such as
``broctl help``)::

  > broctl
  Welcome to BroControl x.y

  Type "help" for help.

  [BroControl] >

As the message says, type help_ to see a list of
all commands. We will now briefly summarize the most important
commands. A full reference follows `Command Reference`_.

The config_ command gives a list of all BroControl options with their
current values.  This can be useful when troubleshooting a problem to
check if an option has the expected value.

If this is the first time you are running BroControl, then the first command
you must run is the BroControl deploy_ command.  The "deploy" command
will make sure all of the files needed by BroControl and Bro are brought
up-to-date based on the configuration specified in the ``broctl.cfg``,
``node.cfg``, and ``networks.cfg`` files.  It will also check if there
are any syntax errors in your Bro policy scripts. For a cluster setup it will
copy all of the required scripts and executables to all the other hosts
in your cluster.  Then it will successively start manager, proxies, and
workers (for a standalone configuration, only one Bro instance will be
started).

The status_ command can be used to check that all nodes are "running".
If any nodes have a status of "crashed", then use the diag_ command to
see diagnostic information (you can specify the name of a crashed node
as an argument to the diag command to show diagnostics for only that one
node).

To stop the monitoring, issue the stop_ command. After all
nodes have stopped, the status_ command should show all nodes as "stopped".
exit_ leaves the shell (you can also exit BroControl while Bro nodes are
running).

Whenever the BroControl or Bro configuration is modified in any way,
including changes to configuration files and site-specific policy
scripts or upgrading to a new version of Bro, deploy_ must
be run (deploy will check all policy scripts, install all needed files, and
restart Bro). No changes will take effect until deploy_ is run.

Log Files
---------

On the manager system (and on the stand-alone system), you find the
current set of (aggregated) logs in ``logs/current`` (which is a
symlink to the corresponding spool directory).
The logs are archived in ``logs/``, by default
once per hour. Log files of workers and proxies are discarded at the
same rotation interval.

If, for some reason, log files are not able to be archived (you would notice
this by seeing a gap in the set of logs in the ``logs/`` directory), you can
search for unarchived logs with a command like this::

    find spool -name "*.log"

If you see any logs in a subdirectory of ``spool/tmp/``, then those were moved
there when Bro previously stopped or crashed.  If Bro is still running,
then there might be rotated logs (i.e., logs with filenames containing a
timestamp) in Bro's working directory.

Additionally, when broctl starts Bro, it creates two files "stdout.log"
and "stderr.log".  These files are sometimes useful to debug problems,
because they might contain error messages from Bro or from the script
which archives log files.


Site-specific Customization
---------------------------

You'll most likely want to adapt the Bro policy to the local
environment and generally site-specific tuning requires writing
local policy scripts.

Sample local policy scripts (which you can edit)
are located in ``share/bro/site``. In the stand-alone setup, a single
file called ``local.bro`` gets loaded automatically.  In the cluster
setup, the same ``local.bro`` gets loaded, followed by one of three
other files: ``local-manager.bro``, ``local-worker.bro``, and
``local-proxy.bro`` are loaded by the manager, workers, and proxy,
respectively.

The recommended way to modify the policy is to use only "@load" directives
in the ``local.bro`` scripts.  For example, you can add a "@load" directive
to load a Bro policy script that is included with Bro but is not loaded
by default.  You can also create custom site-specific
policy scripts in the same directory as the ``local.bro`` scripts, and "@load"
them from one of the ``local.bro`` scripts.  For example, you could create
your own Bro script ``mypolicy.bro`` in the ``share/bro/site`` directory,
and then add a line "@load mypolicy" (without the quotes) to the ``local.bro``
script.  Note that in a cluster setup, notice filtering should be done only
on the manager.

After creating or modifying your local policy scripts, you must install them
by using the broctl "install" or "deploy" command.  Next, you can use the
broctl "scripts" command to verify that your new scripts will be loaded when
you start Bro.

If you want to change which local policy scripts are loaded by the nodes,
you can set SitePolicyStandalone_ for all Bro instances,
SitePolicyManager_ for the manager, and SitePolicyWorker_ for the
workers.  To change the directory where local policy scripts are
located, set the option SitePolicyPath_ to a different path.  These
options can be changed in the ``broctl.cfg`` file.

Load Order of Scripts
~~~~~~~~~~~~~~~~~~~~~

When writing custom site-specific policy scripts, it can sometimes be useful
to know in which order the scripts are loaded (the broctl "scripts" command
shows the load order of every script loaded by Bro).  For example, if more than
one script sets a value for the same global variable, then the value that
takes effect is the one set by the last such script loaded.

When broctl starts Bro, the first script loaded is init-bare.bro, followed
by init-default.bro (keep in mind that each of these scripts loads many
other scripts).

Next, the local.bro script is loaded.  This provides for a common set of
loaded scripts for all nodes.

Next, the "broctl" script package is loaded.  This consists of some standard
settings that broctl needs.

In a cluster setup, one of the following scripts are loaded:
local-manager.bro, local-proxy.bro, or local-worker.bro.

The next scripts loaded are ``local-networks.bro`` and ``broctl-config.bro``.
These scripts are automatically generated by BroControl based on the
contents of the ``networks.cfg`` and ``broctl.cfg`` files.

The last scripts loaded are any node-specific scripts specified with the
option ``aux_scripts`` in ``node.cfg``.  This option can be used to
load additional scripts to individual nodes only.  For example, one could
add a script ``experimental.bro`` to a single worker for trying out new
experimental code.


Miscellaneous
-------------

Mails
~~~~~

BroControl sends three types of mails to the address given in
``MailTo``:

1. When the ``cron`` command runs it performs various tasks (such as
   checking available disk space, expiring old log files, etc.).  If
   any problems occur, a list of those issues will be sent.

2. When the ``cron`` command notices that a node has crashed, it
   restarts it and sends a notification. It may also send a more
   detailed crash report containing information about the crash.

3. If `trace-summary <http://www.bro.org/sphinx/components/trace-summary/README.html>`_
   is installed, a traffic summary is sent each rotation interval. This
   can be disabled by setting ``MailConnectionSummary=0``.

Command Reference
-----------------

The following summary lists all commands supported by BroControl.
All commands may be either entered interactively or specified on the
shell's command line. If not specified otherwise, commands taking
*[<nodes>]* as arguments apply their action either to the given set of
nodes, to the manager node if "manager" is given, to all proxy nodes if
"proxies" is given, to all worker nodes if "workers" is given, or to all
nodes if none are given.

.. include:: commands.rst

Option Reference
----------------

This section summarizes the options that can be set in ``broctl.cfg``
for customizing the behavior of BroControl (the option names are
case-insensitive). Usually, one only needs
to change the "user options", which are listed first. The "internal
options" are, as the name suggests, primarily used internally and set
automatically. They are documented here only for reference.

.. include:: options.rst

Plugins
-------

BroControl provides a plugin interface to extend its functionality. A
plugin is written in Python and can do any, or all, of the following:

    * Perform actions before or after any of the standard BroControl
      commands is executed. When running before the actual command, it
      can filter which nodes to operate or stop the execution
      altogether. When running after the command, it gets access to
      the commands success status on a per-node basis (where applicable).

    * Add custom commands to BroControl.

    * Add custom options to BroControl defined in ``broctl.cfg``.

    * Add custom keys to nodes defined in ``node.cfg``.

A plugin is written by deriving a new class from BroControl class
`Plugin`_. The Python script with the new plugin is then copied into a
plugin directory searched by BroControl at startup. By default,
BroControl searches ``<prefix>/lib/broctl/plugins``; additional directories
may be configured by setting the SitePluginPath_ option. Note that any plugin
script must end in ``*.py`` to be found. BroControl comes with some
example plugins that can be used as a starting point; see
the ``<prefix>/lib/broctl/plugins`` directory.

In the following, we document the API that is available to plugins. A
plugin must be derived from the `Plugin`_ class, and can use its
methods as well as those of the `Node`_ class.

.. include:: plugins.rst

.. _FAQ:

Questions and Answers
---------------------

*Can I use an NFS-mounted partition as the cluster's base directory to avoid the ``rsync``'ing?*
    Yes. BroBase_ can be on an NFS partition.
    Configure and install the shell as usual with
    ``--prefix=<BroBase>``. Then add ``HaveNFS=1`` and
    ``SpoolDir=<spath>`` to ``broctl.cfg``, where ``<spath>`` is a
    path on the local disks of the nodes; ``<spath>`` will be used for
    all non-shared data (make sure that the parent directory exists
    and is writable on all nodes!). Then run ``make install`` again.
    Finally, you can remove ``<BroBase>/spool`` (or link it to <spath>).
    In addition, you might want to keep the log files locally on the nodes
    as well by setting LogDir_ to a non-NFS directory. (Only
    the manager's logs will be kept permanently, the logs of
    workers/proxies are discarded upon rotation.)

*When I'm using the stand-alone mode, do I still need to have ``ssh`` and ``rsync`` installed and configured?*
    No. In stand-alone mode all operations are performed directly on the local
    file system.

*What do I need to do when something in the Bro distribution changes?*
    After pulling from the main Bro git repository, just re-run ``make
    install`` inside your build directory.  It will reinstall all the
    files from the distribution that are not up-to-date. Then do
    ``broctl deploy`` to make sure everything gets pushed out.

*Can I change the naming scheme that BroControl uses for archived log files?*
    Yes, set MakeArchiveName_ to a
    script that outputs the desired destination file name for an
    archived log file. The default script for that task is
    ``<BroBase>/share/broctl/scripts/make-archive-name``, which you
    can use as a template for creating your own version. See
    the beginning of that script for instructions.

*Can BroControl manage a cluster of nodes over non-global IPv6 scope (e.g. link-local)?*
    Yes, set ``ZoneID`` in ``broctl.cfg`` to the zone identifier
    that the BroControl node uses to identify the scope zone
    (the ``ifconfig`` command output is usually helpful, if it doesn't
    show the zone identifier appended to the address with a '%'
    character, then it may just be the interface name).  Then in
    ``node.cfg``, add a ``zone_id`` key to each node section
    representing that particular node's zone identifier and set
    the ``host`` key to the IPv6 address assigned to the node within
    the scope zone.  Most nodes probably have the same ``zone_id``, but
    may not if their interface configuration differs.  See :rfc:`4007` for
    more information on IPv6 scoped addresses and zones.
