Tuning ActiveMQ

Memory limits

The settings for the embedded ActiveMQ broker are found in ${OPENNMS_HOME}/etc/opennms-activemq.xml. Memory and storage limits are conservative by default and should be tuned to accommodate your workload. Consider increasing the memoryUsage (defaults to 20MB) to 512MB or greater, assuming you have enough heap available.

<systemUsage>
    <systemUsage>
        <memoryUsage>
            <memoryUsage limit="20 mb"/> (1)
        </memoryUsage>
        <storeUsage>
            <storeUsage limit="1 gb"/> (2)
        </storeUsage>
        <tempUsage>
            <tempUsage limit="100 mb"/> (3)
        </tempUsage>
    </systemUsage>
</systemUsage>

1	The amount of JVM heap memory assigned to the broker to track of destinations, cache messages, etc; `memoryUsage` limit can be increased to `512MB` or more.
2	The limit of disk storage in `${OPENNMS_HOME}/data/` that can be used to store persistent messages; `storeUsage` limit can be increased to `5GB` or more.
3	The limit of disk storage in `${OPENNMS_HOME}/data/` that can be used to store non-persistent messages; `tempUsage` limit can be increased to `1GB` or more.

If the memory limit is reached, flow control will prevent messages from being published to the broker.

Per destination limits

ActiveMQ provides mechanism to tune memory usage per destination. The default limits are conservative and should be tuned to accommodate your workload.

<policyEntry queue=">" (1)
             producerFlowControl="true"
             memoryLimit="1mb"> (2)
   <deadLetterStrategy>
     <discarding/>
   </deadLetterStrategy>
 </policyEntry>

1	This matches all queues
2	This is the `memoryLimit` per queue and should be increased.

The policyEntry/memoryLimit value prevents any one queue from consuming all broker resources, but should be tuned such that the per-queue limit is a reasonable percentage of the total systemUsage/memoryUsage limit.

KahaDB persistence

KahaDB is a file based persistence database that is local to the ActiveMQ broker. It has been optimized for fast persistence and is the the default broker storage mechanism. ActiveMQ spools messages to disk in order to prevent broker running out of memory or to fulfill persistence requirements.

You can optimize the performance of the KahaDB message store by modifying the following attributes:

<persistenceAdapter>
    <kahaDB directory="${activemq.data}/kahadb"
            journalMaxFileLength="128mb" (1)
            journalDiskSyncStrategy="never" (2)
            indexWriteBatchSize="10000" (3)
     />
</persistenceAdapter>

1	Because there is a cost associated with closing a full journal file and opening a new journal file, you can get a slight performance improvement by increasing the journal file size (Default `32MB`)
2	By default the broker performs an expensive disk sync before sending an acknowledgement back to a producer. A substantial improvement in broker performance can be achieved by disabling these syncs. (Default: `true`)
3	Defines the threshold for the number of dirty indexes that are allowed to accumulate, before KahaDB writes the cache to the store. This consolidates more small journal writes into a single larger write operation. (Default: 1000)

Monitoring the ActiveMQ broker using the Karaf shell

Use the opennms:activemq-stats command available via the Karaf shell to show statistics about the embedded broker:

opennms:activemq-stats

If the command is not available, try installing the feature using feature:install opennms-activemq-shell

This command reports some high level broker statistics as well as message, enqueue and dequeue counts for the available queues. Pay close attention to the memory usage that is reported. If the usage is high, use the queue statistics to help isolate which queue is consuming most of the memory.

The opennms:activemq-purge-queue command can be used to delete all of the available messages in a particular queue:

opennms:activemq-purge-queue OpenNMS.Sink.Trap

Authentication and authorization with ActiveMQ

The embedded ActiveMQ broker is preconfigured to authenticate clients using the same authentication mechanisms (JAAS) as the Meridian web application.

Users associated with the ADMIN role can read, write or create any queue or topic.

Users associated with the MINION role are restricted in such a way that prevents them from making RPC requests to other locations, but can otherwise read or write to the queues they need.

See the authorizationPlugin section in ${OPENNMS_HOME}/etc/opennms-activemq.xml for details.

Multi-tenancy with Meridian and ActiveMQ

The queue names Meridian uses are prefixed with a constant value. If many Meridian are configured to use the same broker, then these queues would end up being shared among the instances, which is not desired. In order to isolate multiple instances on the same broker, you can customize the prefix by setting the value of the org.opennms.instance.id system property to something that is unique per instance.

${OPENNMS_HOME}/etc/opennms.properties.d/instance-id.properties

org.opennms.instance.id=MyNMS

Update the Minion’s instance ID accordingly to match the Meridian instance.

${MINION_HOME}/etc/custom.system.properties.

org.opennms.instance.id=MyNMS

If you change the instance ID setting when using the embedded broker, you will need to update the authorization section in the broker’s configuration to reflect the updated prefix. Modify the configuration with the file ${OPENNMS_HOME}/etc/opennms-activemq.xml:

<authorizationPlugin>
  <map>
    <authorizationMap>
      <authorizationEntries>
        <!-- Users in the admin role can read/write/create any queue/topic -->
        <authorizationEntry queue=">" read="admin" write="admin" admin="admin" />
        <authorizationEntry topic=">" read="admin" write="admin" admin="admin"/>
        <!-- Users in the minion role can write/create queues that are not keyed by location -->
        <authorizationEntry queue="<YOUR INSTANCE ID>.*.*" write="minion" admin="minion" />
        <!-- Users in the minion role can read/create from queues that are keyed by location -->
        <authorizationEntry queue="<YOUR INSTANCE ID>.*.*.*" read="minion" admin="minion" />
        <!-- Users in the minion role can read/write/create advisory topics -->
        <authorizationEntry topic="ActiveMQ.Advisory.>" read="minion" write="minion" admin="minion"/>
      </authorizationEntries>
      <!-- Allow all users to read/write/create temporary destinations (by omitting a <tempDestinationAuthorizationEntry>) -->
    </authorizationMap>
  </map>
</authorizationPlugin>

Tuning the RPC client in OpenNMS

The following system properties can be used to tune the thread pool used to issue RPCs:

Name Default Description

Name	Default	Description
org.opennms.ipc.rpc.threads	10	Number of threads which are always active.
org.opennms.ipc.rpc.threads.max	20	Maximum number of threads which can be active. These will exit after remaining unused for some period of time.
org.opennms.ipc.rpc.queue.max	1000	Maximum number of requests to queue. Set to `-1` to be unlimited.
org.opennms.activemq.client.max-connections	8	The maximum number of concurrent connections to the ActiveMQ broker; larger numbers allow for increased concurrency and throughput.
org.opennms.activemq.client.concurrent-consumers	10	The maximum number of concurrent ActiveMQ consumers to spawn; larger numbers can increase concurrency and throughput.

org.opennms.ipc.rpc.threads

Number of threads which are always active.

org.opennms.ipc.rpc.threads.max

Maximum number of threads which can be active. These will exit after remaining unused for some period of time.

org.opennms.ipc.rpc.queue.max

1000

Maximum number of requests to queue. Set to -1 to be unlimited.

org.opennms.activemq.client.max-connections

The maximum number of concurrent connections to the ActiveMQ broker; larger numbers allow for increased concurrency and throughput.

org.opennms.activemq.client.concurrent-consumers

The maximum number of concurrent ActiveMQ consumers to spawn; larger numbers can increase concurrency and throughput.

Use the opennms:stress-rpc Karaf shell command on the core to help evaluate and tune performance.

Tuning the RPC client in Minion

The following properties can be added to org.opennms.minion.controller.cfg to tune the thread pool used to issue RPCs:

Name	Default	Description
broker-max-connections	8	The maximum number of concurrent connections to the ActiveMQ broker; larger numbers allow for increased concurrency and throughput.
broker-concurrent-consumers	10	The maximum number of concurrent ActiveMQ consumers to spawn; larger numbers can increase concurrency and throughput.

Name

Default

Description

broker-max-connections

The maximum number of concurrent connections to the ActiveMQ broker; larger numbers allow for increased concurrency and throughput.

broker-concurrent-consumers

The maximum number of concurrent ActiveMQ consumers to spawn; larger numbers can increase concurrency and throughput.

Use the opennms:stress-rpc Karaf shell command on the core to help evaluate and tune performance.

Troubleshooting RPC failures

Symptoms of RPC failures may include missed polls, missed data collection attempts and the inability to provision or re-scan existing nodes. For these reasons, it is important to ensure that RPC related communication with Minion at the various monitoring locations remains healthy.

If you want to verify that a specific location is operating correctly make sure that:

Nodes exist and were automatically provisioned for all of the Minions at the location
The Minion-Heartbeat, Minion-RPC and JMX-Minion services are online for one or more Minions at the location
Response time graphs for the Minion-RPC service are populated and contain reasonable values
- These response time graphs can be found under the 127.0.0.1 response time resource on the Minion node
- Values should typically be under 100ms but may vary based on network latency
Resource graphs for the JMX-Minion service are populated and reasonable values

To interactively test RPC communication with a remote location use the opennms:poll command from the Karaf shell:

opennms:poll -l LOCATION -c org.opennms.netmgt.poller.monitors.IcmpMonitor 127.0.0.1 (1)

1	Replace `LOCATION` in the command above with the name of the monitoring location you want to test.