# Monitoring, Tuning and Debugging

UIMA AS deployments can involve many separate parts running on many different machines. Monitoring facilities and tools built into UIMA AS help in collecting information on the performance of these parts. You can use the monitoring information to identify deployment issues, such as bottlenecks, and address these with various approaches that alter the deployment choices; this is what we mean by "tuning the deployment".

Monitoring happens in several parts:

- Each node running a JVM hosting UIMA AS services or clients provides JMX information tracking many items of interest.

- UIMA AS services include some of these measurements in the information passed back to its client, along with the returned CAS. This allows clients to collect and aggregate measurements over a cluster of remotely-deployed components.

Tuning a UIMA AS application is done using several approaches:

- changing the topology of the scaleout - for instance, allocating more nodes to some parts, less to others

- adjusting deployment parameters, such as the number of CASes in a CasPool, or the number of threads assigned to do various tasks

In addition, tuning can involve changing the actual analytic algorithms to tune them - but that is beyond the scope of this chapter.

UIMA AS scale out configurations add multithreaded and out-of-order execution complexities to core UIMA applications. Debugging a UIMA AS application is aided by UIMA's modular architecture and an approach that exercises the code gradually from simpler to more complex configurations. Two useful built-in debug features are:

- Java errors at any component level are propagated back to the component originating the request, with a full call chain of UIMA AS components, within colocated aggregate components and across remote services which are shared by multiple clients.

- CASes can be saved before sending to any local or remote delegate and later used to reproduce problems in a simple unit testing environment.

## Monitoring

### JMX

JMX (Java Management Extensions) is a standard Java mechanism that is used to monitor and control Java applications. A standard Java tool provided with most Javas, called `jconsole`, is a GUI based application that can connect to a JVM and display the information JMX is providing, and also control the application in application-defined specific ways.

JMX information is provided by a hierarchy of JMX Beans. More background and information on JMX and the jconsole tool is available on the web.

### JMX Information from UIMA AS

JMX information is provided by every UIMA AS service or client as it runs. Each item provided is either an instantaneous measurement ( e.g. the number of items in a queue) or an accumulating measurement ( e.g. the number of CASes processed). Accumulating measures can be reset to 0 using standard JMX mechanisms.

JMX information is provided on a JVM basis; a JVM can be hosting 0 or more UIMA AS Services and/or clients. A UIMA AS Service is defined as a component that connects to a queue and accepts CASes to process. A UIMA AS Client, in contrast, sends CASes to be processed; it can be a top level client, or a UIMA AS Service having one or more AS Aggregate delegates, to which it is sending CASes to be processed.

UIMA AS Services send some of their measurements back to the UIMA AS Clients that sent them CASes; those clients incorporate these measurements into aggregate statistics that they provide. This allows accumulating information among components deployed over many nodes interconnected on a network.

Some JMX measurement items are constant, and document various settings, descriptors, names, etc., in use by the (one or more) UIMA AS services and/or clients running on this JVM.

Some time measurements are associated with running some process. These, where possible, are cpu times, as measured by the thread or threads running the process, using the ThreadMXBean class. On some Javas, thread-based cpu time may not be supported, however. In that case, wall-clock time is used instead.

If the process is multi-threaded, and the cpu has multiple cores, you can get time measurements which exceed the wall clock interval, due to the process consuming cpu time on multiple threads at once.

Timing information not associated with running code, such as idle time, is measured as wall-clock time.

The following sections describe the JMX Beans implemented by UIMA AS. The Notes in the tables include the following flags:

- inst/acc/const - instantaneous, accumulating, or constant measurement

- sent - sent up to the invoking client with returning CAS

#### UIMA AS Services JMX measures

The next 4 tables detail the JMX measures provided by UIMA AS services.

##### Service information

Name | Description | Units | Notes
--- | --- | --- | ---
state | The state of the service (Running, Initializing, Disabled, Stopping, Failed) | string | inst
input queueName | The name of the input queue | string | const
reply queueName | The internally generated name of the reply queue | string | const (but could change due to reconnection recovery)
broker URL | The URL of the JMS queue broker | string | const
deployment descriptor | The path to the deployment descriptor for this service | string | const
is CAS Multiplier | is this Service a CAS Multiplier | boolean | const
is top level | is this Service a top level service, meaning that it connects to an input queue on a queue broker | boolean | const
service key | The key name used in the associated Analysis Engine aggregate that specifies this as a delegate | string | const
is Aggregate | is this service an AS Aggregate (i.e., has delegates and is marked async="true") | boolean | const
analysisEngine instance count | The number of replications of the AS Primitive | count | const

##### Service Performance Measurements

Name | Description | Units | Notes
--- | --- | --- | ---
number of CASes processed | The number of CASes processed by a component | count - CASes | acc
cas deserialization time | The thread time spent deserializing CASes (receiving, either from client, or replies from delegates) | milli seconds | acc
cas serialization time | The thread time spent serializing CASes (sending, either to delegates or back to client) | count - CASes | acc
analysis time | The thread time spent in AS Primitive analytics | milli seconds | acc
idle time | The wall clock time a service has been idle. Measure starts after a reply is sent until the next request is receives, and excludes serialization/deserialization times. | milli seconds | acc
cas pool wait time | The time spent waiting for a CAS to become available in the CAS Pool | milli seconds | acc
shadow cas pool wait time | A shadow cas pool is established for services which are Cas Multipliers. This is the time spent waiting for a CAS to become available in the Shadow CAS Pool. | milli seconds | acc
time spent in CM getNext | The time spent inside Cas Multipliers, getting another CAS. This time (doesn't include / includes ????) the time spent waiting for a CAS to become available in the CAS Pool waiting for a CAS to become available in the CAS Pool | milli seconds | acc
process thread count | The number of threads available to process requests (number of instances of a primitive) | count | const
reply thread count | The number of threads available to process replies | count | const

##### Co-located Service Queues

Co-located services use light-weight, internal (not JMS) queues. These have similar measures as are used with JMS queues, and include these measures for both the input queues and the reply (output) queues:

Name | Description | Units | Notes
--- | --- | --- | ---
consumer count | The number of threads configured to read the queue | count | const
dequeue count | The number of CASes that have been read from this queue | count | acc
queue size | The number of CASes in the queue | count | inst

##### Service Error Measurements

Name | Description | Units | Notes
--- | --- | --- | ---
process Errors | The number of process errors | count | acc
getMetadata Errors | The number of getMetadata errors | count | acc
cpc Errors | The number of Collection Process Complete (cpc) errors | count | acc

#### Application Client information

This section describes monitoring information provided by the UIMA AS Client APIs. Any code that uses the , such as the example application client `RunRemoteAsyncAE`, will have a set of these JMX measures. Currently no additional tooling (beyond standard tools like `jconsole`) are provided to view these.

##### Client Measures

Name | Description | Units | Notes
--- | --- | --- | ---
application Name | A user-supplied string identifying the application | string | const
service queue name | The name of the service queue this client connects to | string | const
serialization method | either xmi or binary. This is the serialization the client will use to send CASes to the service, and also tells the service which serialization to use in sending the CASes back. | string | const
cas pool size | This client's cas pool size, limiting the number of simultaneous outstanding requests in process | count | const
total number of CASes processed | count of the total number of CASes sent from this client. Note: in the case where the service is a Cas Multiplier, the "child" CASes are not included in this count. | count | acc
total time to process | total thread time spent in processing all CASes, including time in remote delegates | milli seconds | acc
average process time | total number of CASes processed / total time to process | milli seconds | inst
max process time | maximum thread time spent in processing a CAS, including time in remote delegates | milli seconds | inst
total serialization time | total thread time spent in serializing, both to delegates (and recursively, to their delegates) and replies back to senders | milli seconds | acc
average serialization time | average thread time spent in serializing a CAS, both to delegates (and recursively, to their delegates) and replies back to senders | milli seconds | inst
max serialization time | maximum thread time spent in serializing a CAS, both to delegates (and recursively, to their delegates) and replies back to senders | milli seconds | inst
total deserialization time | total thread time spent in deserializing, both replies from delegates and CASes from upper level components being sent to lower level ones. | milli seconds | acc
average deserialization time | average thread time spent in deserializing, both replies from delegates and CASes from upper level components being sent to lower level ones. | milli seconds | inst
max deserialization time | maximum thread time spent in deserializing, both replies from delegates and CASes from upper level components being sent to lower level ones. | milli seconds | inst
total idle time | total wall clock time a top-level service thread has been idle since the thread was last used. If there is more than one service thread, this number is the sum. | milli seconds | acc
average idle time | average wall clock time all top-level service threads have been idle since they were last used | milli seconds | inst
max idle time | maximum wall clock time a top-level service thread has been idle since the thread was last used | milli seconds | inst
total time waiting for reply | total wall clock time, measured from the time a CAS is sent to the top-level queue, until that CAS is returned. Any generated CASes from Cas Multipliers are not counted in this measurement. | milli seconds | acc
average time waiting for reply | average wall clock time from the time a CAS is sent to the reply is received | milli seconds | inst
max time waiting for reply | maximum wall clock time from the time a CAS is sent to the reply is received | milli seconds | inst
total response latency time | total wall clock time, measured from the time a CAS is sent to the top-level queue, including the serialization and deserialization times at the client, until that CAS is returned. Any generated CASes from Cas Multipliers are not counted in this measurement. | milli seconds | acc
average response latency time | average wall clock time, measured from the time a CAS is sent to the top-level queue, including the serialization and deserialization times at the client, until that CAS is returned. | milli seconds | inst
max response latency time | maximum wall clock time, measured from the time a CAS is sent to the top-level queue, including the serialization and deserialization times at the client, until that CAS is returned. | milli seconds | inst
total time waiting for CAS | total wall-clock time spent waiting for a free CAS to be available in the client's CAS pool, before sending the CAS to input queue for the top level service. | milli seconds | acc
average time waiting for CAS | average wall-clock time spent waiting for a free CAS to be available in the client's CAS pool | milli seconds | inst
max time waiting for CAS | maximum wall-clock time spent waiting for a free CAS to be available in the client's CAS pool | milli seconds | inst
total number of CASes requested | total number of CASes fetched from the CAS pool | count | acc

##### Client Error Measurements

Name | Description | Units | Notes
--- | --- | --- | ---
getMeta Timeout Error Count | number of times a getMeta timed out | count | acc
getMeta Error Count | number of times a getMeta request returned with an error | count | acc
process Timeout Error Count | number of times a process call timed out | count | acc
process Error Count | number of times a process call returned with an error | count | acc

## Logging Sampled JMX information at intervals

A common tuning procedure is to run a deployment for a fairly long time with a typical load, and to see what and where hot spots develop. During this process, it is sometimes useful to convert accumulating measurements into averages, perhaps averages per CAS processed.

UIMA AS includes a monitor component, org.apache.uima.aae.jmx.monitor.JmxMonitor, to sample JMX measures at specified intervals, compute various averages, and write the results into the UIMA Log (or on the console if no log is configured). The monitor program can be automatically enabled for any deployed service by specifying `-D` parameters on the JVM command line which launches the service, or, it can be run stand-alone; when run stand-alone, you provide an argument specifying the JVM it is to connect to to get the JMX information. It only connects to one JVM per run; typically, you would connect it to the top-level service.

The monitor outputs information for that service and its immediate delegates (local or remote); however, it includes information from the complete recursive chain of delegates when computing its measures. You can get detailed monitoring for sub-services by starting or attaching a monitor to those sub-services.

ActiveMQ uses Queue Brokers to manage the JMS queues used by UIMA AS. These brokers have JMX information that is useful in tuning applications. The Monitor program identifies the Queue Broker being used by the service, and connects to it and incorporates information about queue lengths (both the input queue and the reply queue) into its measurements.

### Configuring JVM to run the monitor

Specify the following JVM System Variable parameters to configure a UIMA AS Client or Service to enable sampling and logging of JMX measures:

- `-Duima.jmx.monitor.interval=1000` - (default is 1000) specifies the sampling interval in milliseconds

- `-Duima.jmx.monitor.formatter=<CustomFormatterClassName>`

- `-Dcom.sun.management.jmxremote` - enable JMX (only needed for local monitoring, not needed if port is specified)

- `-Dcom.sun.management.jmxremote.port=8009`

- `-Dcom.sun.management.jmxremote.authenticate=false`

- `-Dcom.sun.management.jmxremote.ssl=false`

This configures JMX to run on port 8009 with no authentication, and sets the sampling interval to 1 second, and specifies a custom formatter class name.

There are two `formatter-classes` provided with UIMA AS:

- `org.apache.uima.aae.jmx.monitor.BasicUimaJmxMonitorListener -` this is a multi-line formatter that formats for human-readable output

- `org.apache.uima.aae.jmx.monitor.SingleLineUimaJmxMonitorListener -` this is a formatter that produces one line per interval, suitable for importing into a spreadsheet program.

Both of these log to the UIMA log at the INFO log level.

You can also write your own formatter. The monitor provides an API to plug in a custom formatter for displaying service metrics. A custom formatter must implement JmxMonitorListener interface. See the method `startMonitor` in the class `UIMA_Service` for an example of how custom JMX Listeners are plugged into the monitor.

### Running the Monitor program standalone

The monitor program can be started separately and pointed to a running UIMA AS Client or Service. To start the program, invoke Java with the following classpath and parameters:

- ClassPath:

  - %UIMA_HOME%/lib/uimaj-as-activemq.jar

  - %UIMA_HOME%/lib/uimaj-as-core.jar

  - %UIMA_HOME%/lib/uima-core.jar

  - %UIMA_HOME%/apache-activemq-5.4.1/activemq-all-5.4.1.jar

- Parameters:

  - `-Djava.util.logging.config.file=%UIMA_HOME%/config/MonitorLogger.properties` - specifies the logging file where the information is written to

  - `org.apache.uima.aae.jmx.monitor.JmxMonitor` - the class whose main method is invoked

  - `uri` - the URI of the jmx instance to monitor.

  - `interval` - the (optional) sampling interval, in milliseconds (default = 1000)

When run in this manner, it is not (currently) possible to specify the log message formatting class; the multi-line output format is always used.

### Monitoring output

The monitoring program combines information from the JMX measures, including the associated Queue Broker, sampling accumulating measurements at the specified sampling interval, and produces the following outputs:

Name | Description | Units>
--- | --- | ---
Input queue depth | number of CASes waiting to be processed by a service | count
Reply queue depth | number of CASes returned to the client but not yet picked up by the client | count
CASes processed in interval | Number of CASes processed in this sampling interval | count
Idle time in interval | The total time this service has been idle during this interval | milli seconds
Analysis time in interval | The sum of the times spent in analysis by the service during this interval, including analysis time spent in delegates, recursively | milli seconds
Cas Pool free Cas Count | Number of available CASes in the Cas Pool at the end of the interval | count

In addition to the performance metrics the monitor also provides basic service information:

- Service name

- Is service top level

- Is service remote

- Is service a cas multiplier

- Number of processing threads

- Service uptime (milliseconds)

## Tuning

### Tuning procedure

This section is a cookbook of best practices for tuning a UIMA AS deployment. The summary information provided by the Monitor program is used to guide the tuning.

The main metric for detecting an overloaded service is the input queue depth. If it is growing or high, the service is not able to keep up with the load. There are more CASes arriving at the queue than the service can process. Consider increasing number of instances of the services within the JVM (if on a multi-core machine having additional capacity), or deploy additional instances of the service.

The main metric for detecting idle service is the idle time. If it is high, it can indicate that the service is not receiving enough CASes. This can be caused by a bottleneck in the service's client; supporting evidence for this can be a high reply queue depth for the client - indicating the client is overloaded. If the idle time is zero, the service may be saturated; adding more instances could relieve a bottleneck.

A CasPool free Cas Count of 0 can point to a bottleneck in a service's client; supporting evidence for this can be a high idle time. In this case, the service does not have enough CASes in its pool and is forced to wait. Remember that a CAS is not returned to the Service's CAS pool until the client (which can be a parent asynchronous aggregate) signals it can be. A typical reason is a slow client (look for evidence such as a high reply queue depth). Consider incrementing service's Cas pool and check the client's metrics to determine a reason why it is slow.

An asynchronous system must have something that limits the generation of new work to do. CasPools are the mechanism used by UIMA AS to do this. Also, because CASes can have large memory requirements, it is important to limit the number and sizes of CASes in a process.

### Tuning Settings

This section has a list of the tuning parameters and a description of what they do and how they interact.

Name | Description
--- | ---
number of services on different machines started | You can adjust the number of machines assigned to a particular service, even dynamically, by just starting / stopping additional servers that specify the same input queue.
number of instances of a service | This is similar to the number of services on different machines started, above, but specifies replication of an AS Primitive within one JVM. This is useful for making use of multi-core machines sharing a common memory - large tables that might be part of the analysis algorithm can be shared by all instances.
CAS pool size | This size limits the number of CASes being processed asynchronously.
casMultiplier poolSize | This size limits the number of CASes generated by a CAS Multiplier that are being processed asynchronously.
Service input queue prefetch | If set greater than 0, allows up to "n" CASes to be pulled into one service provider, at a time. This can increase throughput, but can hurt latency, since one service may have several CASes pulled into it, queued up, while another instance of the service could be "starved" and be sitting there idle.
Specifying async="true"/"false" on an aggregate | The default is false, because there is less overhead (no queues are set up, etc.). Setting this to "true" allows multiple CASes to flow simultaneously in the aggregate.
remoteReplyQueueScaleout | This parameter indicates the number of threads that will be deployed to read from the remote reply queue. Set to > 1 if deserialization time of replies is a bottleneck.

## Debugging

One of the strongest UIMA features is the ability to develop and debug components in isolation from each other, and then to incrementally combine components and scaleout complexity. All that is needed to exercise each configuration are one or more appropriate input CASes.

It is strongly advised to first test UIMA components in the core UIMA environment with a variety of input CASes. If the entire application will not fit in a single process, deploy remote delegates as UIMA AS primitives with only a single instance (see ), and access them via JMS service descriptors (see ). Run as much input data thru this "single-threaded" configuration as needed to eliminate most "algorithmic" errors and to measure performance against analysis time objectives. Thread safety and analysis ordering issues can then be addressed separately.

Thread safety bugs. Components intended to be run multi-threaded should first be deployed as a multiple instance UIMA AS service (again see ), and fed their input CASes with a driver capable of keeping all instances busy at the same time. A good application is the sample driver $UIMA_HOME/bin/runRemoteAsyncAE; use the -p argument to increase the number of outstanding CAS requests sent to the target service. When looking for threading problems try using http://findbugs.sourceforge.net/. In addition to looking for exceptions caused by thread unsafe code, check that the single and multi-threaded analysis results are the same.

Analysis ordering bugs. In a core UIMA aggregate CASes are processed by each delegate in input order. This relationship changes for the same aggregate deployed asynchronously if one of the delegates is replicated, as CASes are progressed in parallel and then progress thru the subsequent aggregate flow in a different order then they are received. Similarly with a delegate CasMultiplier in a core UIMA aggregate each child CAS is processed to completion before the next child CAS is started and the parent CAS is processed last. When running asynchronously the parent CAS can arrive at downstream components ahead of its children because the parent is released from a CasMultiplier immediately after the last child is created. For applications which require all children to be processed before their parent, use the processParentLast flag (see ).

Timing issues. Invariably with complex analytics, some components will be slower and some artifacts will take longer to process than desired. Making performance improvements relies on identifying components running slower than expected and capturing the slow-running artifacts to study in detail.

### Error Reporting and Tracing

After the system is scaled out and substantially more data is being processed it is likely that additional errors will occur.

Java errors at any component level are propagated back to the component originating the request (unless suppressed by UIMA AS error handling options, see ). The error stack traces the call chain of UIMA AS components, within colocated aggregate components and across remote services which are shared by multiple clients. Some errors can be resolved with this information alone.

If process timeouts are not used (see ) an asynchronous system can hang if one analysis step somewhere in the system has hung. Given many CASes in process at the same time it can be useful to create a custom trace of CAS activity by appropriate logging in a custom flow controller. Such logging would have a unique identifier in every CAS, usually a singleton FeatureStructure with a unique String feature. Identifiers for child CASes should include some reference to the CasMultiplier they were created from as well as their parent CAS.

The flow controller is also the ideal place to measure timing statistics for components of interest. Global stats can easily be measured using the time between flow steps, and time thresholds used to flag specific CASes causing problems. Again the unique CAS identifier can be quite useful here.

### CAS Logging

Within a UIMA AS asynchronous aggregate, CASes can be saved before sending to any local or remote delegate and later used to reproduce a problem in a simple unit testing environment. Control of CAS logging is done via Java properties:

Property | Description
--- | ---
UIMA_CASLOG_BASE_DIRECTORY | optional; this is the directory under which sub-directories with XmiCas files will be created. If not specified, the process's current directory will be the base.
UIMA_CASLOG_COMPONENT_ARRAY | This is a space separated list of delegates keys. If a delegate is nested inside a co-located async aggregate, the name would include the key name of the aggregate, e.g. "someAggName/someDelName". The XmiCas files will then be written into $UIMA_CASLOG_BASE_DIRECTORY/someAggName-someDelName/
UIMA_CASLOG_TYPE_NAME | optional; this is the name of a FeatureStructure in the CAS containing a unique string to use to name each XmiCas file. If not specified, XmiCas file name will be N.xmi, where N is the time in microseconds since the component was initialized.
UIMA_CASLOG_FEATURE_NAME | optional unless the TYPE_NAME is specified; this parameter gives the string feature to use. If the string value contains one or more "/" characters only the text after the last "/" will be used.
UIMA_CASLOG_VIEW_NAME | optional; if the TYPE_NAME and FEATURE_NAME parameters are specified this string selects the CAS view used to access the FeatureStructure with unique string feature.