Sentinel Features

The following list contains some features which may be installed manually:

Feature Required Description

sentinel-core

true

Base feature, installing all required bundles such as opennms:health-check and service requirements for other bundles, e.g. sentinel-persistence.

sentinel-jms

false

Provides connectivity to the Horizon ActiveMQ Broker.

sentinel-kafka

false

Provides connectivity to Kafka.

sentinel-flows

false

Feature which starts all dependencies to start processing flows.

sentinel-newts

false

Provides functionality to persist measurement data to Newts.

sentinel-telemetry-nxos

false

Lets you use NxosGpbAdapter

sentinel-telemetry-jti

false

Lets you use the JtiGpbAdapter

sentinel-telemetry-bmp

false

Lets you use the BmpTelemetryAdapter

Auto install

In some cases it is desired to automatically configure the Sentinel instance and also start required features/bundles. As Sentinel is based on Apache Karaf - which supports auto deployment by simply copying any kind of data to the deploy folder, Sentinel can make use of that mechanism to enable auto or hot deployment.

In order to do so, in most cases it is sufficient to copy a features.xml file to ${SENTINEL_HOME}/deploy. This can be done even if the container is running.

The chapter Configure Flow Processing contains an example on how to automatically start them with Sentinel

Auto Start

In some cases it might not be sufficient to auto-deploy/configure the container with a features.xml file. If more flexibility is required it is suggested to modify/copy *.cfg and *.properties files directly to the ${SENTINEL_HOME}/etc directory. To automatically start features with the container, create a file in ${SENTINEL_HOME}/etc/featuresBoot.d/ that contains the desired features:

echo "sentinel-jms" | sudo tee -a \${SENTINEL_HOME}/etc/featuresBoot.d/features.boot (1)
echo "sentinel-flows" | sudo tee -a ${SENTINEL_HOME}/etc/featuresBoot.d/sentinel-features.boot (2)
1 Install and start JMS communication feature.
2 Install and start Sentinel flows feature.

Auto configure flow processing for Sentinel

The following examples illustrate a features.xml which configures the Sentinel instance and automatically starts all required features to either consume messages via JMS (ActiveMQ) or Kafka.

Simply copy it to ${SENTINEL_HOME}/deploy/.

JMS
<?xml version="1.0" encoding="UTF-8"?>
<features
        name="opennms-${project.version}"
        xmlns="http://karaf.apache.org/xmlns/features/v1.4.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://karaf.apache.org/xmlns/features/v1.4.0 http://karaf.apache.org/xmlns/features/v1.4.0"
>
    <!-- Bootstrap feature to start all flow related features automatically -->
    <feature name="autostart-sentinel-flows" version="${project.version}" start-level="100" install="auto">
        <!-- Configure the controller itself -->
        <config name="org.opennms.sentinel.controller">
            location = SENTINEL
            id = 00000000-0000-0000-0000-000000ddba11
            broker-url = failover:tcp://127.0.0.1:61616
        </config>

        <!-- Configure datasource connection -->
        <config name="org.opennms.netmgt.distributed.datasource">
            datasource.url = jdbc:postgresql://localhost:5432/opennms
            datasource.username = postgres
            datasource.password = postgres
            datasource.databaseName = opennms
        </config>
        <!--
            Starts the Netflow5Adapter to process Netflow5 Messages.
            Be aware, that this requires a Listener with name "Netflow-5" on the Minion-side to have messages
            processed properly.
        -->
        <config name="org.opennms.features.telemetry.adapters-netflow5">
            name = Netflow-5
            class-name = org.opennms.netmgt.telemetry.adapters.netflow.v5.Netflow5Adapter
        </config>
        <!-- Point sentinel to the correct elastic endpoint -->
        <config name="org.opennms.features.flows.persistence.elastic">
            elasticUrl = http://elasticsearch:9200
        </config>
        <!-- Install JMS related features -->
        <feature>sentinel-jms</feature>
        <!-- Install Flow related features -->
        <feature>sentinel-flows</feature>
    </feature>
</features>
Kafka
<?xml version="1.0" encoding="UTF-8"?>
<features
        name="opennms-${project.version}"
        xmlns="http://karaf.apache.org/xmlns/features/v1.4.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://karaf.apache.org/xmlns/features/v1.4.0 http://karaf.apache.org/xmlns/features/v1.4.0"
>
    <!-- Bootstrap feature to start all flow related features automatically -->
    <feature name="autostart-sentinel-telemetry-flows" version="${project.version}" start-level="200" install="auto">
        <!-- Configure the controller itself -->
        <config name="org.opennms.sentinel.controller">
            location = SENTINEL
            id = 00000000-0000-0000-0000-000000ddba11
            broker-url = failover:tcp://127.0.0.1:61616
        </config>

        <!-- Configure datasource connection -->
        <config name="org.opennms.netmgt.distributed.datasource">
            datasource.url = jdbc:postgresql://localhost:5432/opennms
            datasource.username = postgres
            datasource.password = postgres
            datasource.databaseName = opennms
        </config>
        <!--
            Starts the Netflow5Adapter to process Netflow5 Messages.
            Be aware, that this requires a Listener with name "Netflow-5" on the Minion-side to have messages
            processed properly.
        -->
        <config name="org.opennms.features.telemetry.adapters-netflow5">
            name = Netflow-5
            class-name = org.opennms.netmgt.telemetry.adapters.netflow.v5.Netflow5Adapter
        </config>
        <!-- Point sentinel to the correct elastic endpoint -->
        <config name="org.opennms.features.flows.persistence.elastic">
            elasticUrl = http://elasticsearch:9200
        </config>
        <!--
            Configure as Kafka Consumer.
            All properties described at https://kafka.apache.org/0100/documentation.html#newconsumerconfigs are supported.
        -->
        <config name="org.opennms.core.ipc.sink.kafka.consumer">
            group.id = OpenNMS
            bootstrap.servers = localhost:9092
        </config>
        <!--
            Configure as Kafka Producer for sending Events from Sentinel.
            All properties described at https://kafka.apache.org/0100/documentation.html#producerconfigs are supported.
        -->
        <config name="org.opennms.core.ipc.sink.kafka">
            bootstrap.servers = localhost:9092
        </config>
        <!-- Install Kafka related features -->
        <feature>sentinel-kafka</feature>
        <!-- Install flow related features -->
        <feature>sentinel-flows</feature>
    </feature>
</features>

Kafka Configuration

Each Minion works as a producer and must be configured beforehand. Please refer to section Minion Kafka Producer Configuration on how to configure Minion as a Kafka Producer.

Each Sentinel works as a Consumer and can be configured in the file ${SENTINEL_HOME}/etc/org.opennms.core.ipc.sink.kafka.consumer.cfg. Either manually or via the config:edit org.opennms.core.ipc.sink.kafka.consumer statement. For supported properties, see here

By default each Kafka Consumer starts consuming messages immediately after the feature has been started. It is possible to set a property org.opennms.core.ipc.sink.initialSleepTime to define an initial sleep time in ms before any messages are consumed. In order to set this up, please add an entry to the end of the file ${SENTINEL_HOME}/etc/system.properties:

# Initial delay of 5 seconds before consuming of messages is started in milliseconds
org.opennms.core.ipc.sink.initialSleepTime=5000

Persisting Collection Sets to Newts

In the previous chapter it is described on how to set up Horizon, Minion and Sentinel in order to distribute the processing of flows. However, it only covered flow processing adapters, but there are more, e.g. the NxosGpbAdapter, which can also be run on a Sentinel.

The configuration of Newts for Sentinel uses the same properties as for Horizon. The only difference is, that the properties for Sentinel are stored in /opt/sentinel/etc/org.opennms.newts.config.cfg instead of *.properties files. The name of each property is the same as for Horizon without the org.opennms.newts.config prefix. The following example shows a custom Newts configuration using the Sentinel's Karaf Shell.

$ ssh -p 8301 admin@localhost
admin@sentinel> config:edit org.opennms.newts.config
admin@sentinel> config:property-set hostname localhost
admin@sentinel> config:property-set port 9042
admin@sentinel> config:property-set cache.strategy org.opennms.netmgt.newts.support.GuavaSearchableResourceMetadataCache
admin@sentinel> config:update

Adapters

This chapter describes the various adapters which may contain sample data which may be stored to a Persistence Storage and can also run on a Sentinel. At the moment only Newts is supported as a Persistence Storage. See chapter [ga-sentinel-configure-newts] on how to configure Newts.

In order to get it to work properly, please note, that an appropriate listener on the Minion must also be configured. The name of the listener should share the same name on Sentinel.

SFlowTelemetryAdapter

In order to use this adapter, the feature sentinel-flows and sentinel-newts must be installed. In addition either sentinel-jms or sentinel-kafka should be installed and configured properly. See the previous Flow Processing chapter for more details.

If only sample data should be persisted, the following commands can be run on the Sentinel's Karaf Shell

$ ssh -p 8301 admin@localhost
admin@sentinel> config:edit --alias sflow --factory org.opennms.features.telemetry.adapters
admin@sentinel> config:property-set name SFlow-Telemetry
admin@sentinel> config:property-set class-name org.opennms.netmgt.telemetry.adapters.netflow.sflow.SFlowTelemetryAdapter
admin@sentinel> config:property-set parameters.script  /opt/sentinel/etc/sflow-host.groovy
admin@sentinel> config:update

If SFlow flows and the sample data should be processed, multiple adapters can be configured:

config:edit --alias sflow-telemetry --factory org.opennms.features.telemetry.adapters
config:property-set name SFlow
config:property-set adapters.1.name SFlow-Adapter
config:property-set adapters.1.class-name org.opennms.netmgt.telemetry.adapters.netflow.sflow.SFlowAdapter
config:property-set adapters.2.name SFlow-Telemetry
config:property-set adapters.2.class-name org.opennms.netmgt.telemetry.adapters.netflow.sflow.SFlowTelemetryAdapter
config:property-set adapters.2.parameters.script /opt/sentinel/etc/sflow-host.groovy
config:update

Please note, that in both cases the file /opt/sentinel/etc/sflow-host.groovy must be provided manually, e.g. by manually copying it over from Horizon.

NxosGpbAdapter

In order to use this adapter, the feature sentinel-telemetry-nxos and sentinel-newts must be installed. In addition either sentinel-jms or sentinel-kafka should be installed and configured properly. See the previous Flow Processing chapter for more details.

Besides this, configuration files from Horizon must be copied to Sentinel to /opt/sentinel/etc. The following files and directories are required:

  • ${OPENNMS_HOME}/etc/datacollection

  • ${OPENNMS_HOME}/etc/datacollection-config.xml

  • ${OPENNMS_HOME}/etc/resource-types.d

Afterwards the adapter can be set up:

$ ssh -p 8301 admin@localhost
admin@sentinel> config:edit --alias nxos --factory org.opennms.features.telemetry.adapters
admin@sentinel> config:property-set name NXOS
admin@sentinel> config:property-set class-name org.opennms.netmgt.telemetry.protocols.nxos.adapter.NxosGpbAdapter
admin@sentinel> config:property-set parameters.script /opt/sentinel/etc/cisco-nxos-telemetry-interface.groovy
admin@sentinel> config:update

Please note, that the file /opt/sentinel/etc/cisco-nxos-telemetry-interface.groovy must also be provided manually, e.g. by manually copying it over from Horizon.

JtiGpbAdapter

In order to use this adapter, the feature sentinel-telemetry-jti and sentinel-newts must be installed. In addition either sentinel-jms or sentinel-kafka should be installed and be configured properly. See the previous Flow Processing chapter for more details.

Besides this, configuration files from Horizon must be copied to Sentinel to /opt/sentinel/etc. The following files and directories are required:

  • ${OPENNMS_HOME}/etc/datacollection

  • ${OPENNMS_HOME}/etc/datacollection-config.xml

  • ${OPENNMS_HOME}/etc/resource-types.d

Afterwards the adapter can be set up:

$ ssh -p 8301 admin@localhost
admin@sentinel> config:edit --alias jti --factory org.opennms.features.telemetry.adapters
admin@sentinel> config:property-set name JTI
admin@sentinel> config:property-set class-name org.opennms.netmgt.telemetry.protocols.jti.adapter.JtiGpbAdapter
admin@sentinel> config:property-set parameters.script /opt/sentinel/etc/junos-telemetry-interface.groovy
admin@sentinel> config:update

Please note, that the file /opt/sentinel/etc/junos-telemetry-interface.groovy must also be provided manually, e.g. by manually copying it over from Horizon.

Configure Sentinel container

The Sentinel container image reads its configuration from environment variables at startup. For settings not covered by environment variables, you can overlay individual configuration files directly (see Directory conventions).

Values from environment variables are written so that Karaf reads them back unchanged, including backslashes and other special characters. The exception is ${, which Karaf always expands as a placeholder; the container logs a warning when a value contains it.

Settings are re-applied on every start. If you remove a variable that the container wrote to a configuration file, the setting is reset on the next start: the image default is restored if the image ships one, otherwise the property is removed. The container tracks these settings in ${SENTINEL_HOME}/etc/.container-env-keys.

Controller configuration

Variable Description Default value

SENTINEL_ID

Unique identifier for this Sentinel instance. Written to the configuration on first start and kept afterwards, so changing or removing it later has no effect.

generated UUID

SENTINEL_LOCATION

Location name associated with this Sentinel instance.

SENTINEL

OPENNMS_BROKER_URL

ActiveMQ broker URL.

tcp://127.0.0.1:61616
You cannot supply credentials via environment variables without exposing them in docker inspect output and container logs. To store credentials securely, initialize a keystore file using the -s flag and mount it to ${SENTINEL_HOME}/etc/scv.jce.
Table 1. Credentials (use only in non-production environments)
Variable Description

OPENNMS_HTTP_USER

Username for the OpenNMS HTTP API.

OPENNMS_HTTP_PASS

Password for the OpenNMS HTTP API.

OPENNMS_BROKER_USER

Username for ActiveMQ authentication.

OPENNMS_BROKER_PASS

Password for ActiveMQ authentication.

IPC strategy

Set SENTINEL_IPC to choose the inter-process communication transport. The value is not case-sensitive, and the container refuses to start if it is not one of the values below.

Value Description

(unset)

Use Apache Kafka if KAFKA_IPC_BOOTSTRAP_SERVERS is set, otherwise JMS (ActiveMQ), unless a file in featuresBoot.d already installs the transport (see below).

jms

Use JMS (ActiveMQ), even if Kafka variables are set.

kafka

Use Apache Kafka. Requires additional Kafka variables (see Kafka configuration).

The container installs the matching sentinel-jms or sentinel-kafka feature on every start.

If neither SENTINEL_IPC nor KAFKA_IPC_BOOTSTRAP_SERVERS is set and a file in featuresBoot.d (including one provided through an overlay) installs sentinel-jms or sentinel-kafka, the container leaves the IPC choice to that file. Do not combine such a file with either variable: exclusions such as !sentinel-jms apply across all files in featuresBoot.d, so contradicting selections exclude both transports. The container logs a warning when a file excludes the transport selected by the variables.

Kafka configuration

Environment variables prefixed with KAFKA_IPC_ are written to both ${SENTINEL_HOME}/etc/org.opennms.core.ipc.sink.kafka.cfg and ${SENTINEL_HOME}/etc/org.opennms.core.ipc.sink.kafka.consumer.cfg. The prefix is stripped, the remainder is lowercased, and underscores are converted to dots before being written as property keys.

Variable Description Example value

KAFKA_IPC_BOOTSTRAP_SERVERS

Comma-separated list of Kafka broker addresses.

kafka-1:9092,kafka-2:9092

KAFKA_IPC_GROUP_ID

Kafka consumer group ID.

OpenNMS

KAFKA_IPC_*

Any additional Kafka client property.

KAFKA_IPC_AUTO_OFFSET_RESET=latest

Datasource configuration

Variable Description Default value

POSTGRES_HOST

PostgreSQL host.

localhost

POSTGRES_PORT

PostgreSQL port.

5432

POSTGRES_DB

PostgreSQL database name.

opennms

POSTGRES_USER

PostgreSQL username.

postgres

POSTGRES_PASSWORD

PostgreSQL password.

–

POSTGRES_JDBC_URL

Complete JDBC URL, for example jdbc:postgresql://db:5432/opennms?sslmode=require. Replaces the URL built from POSTGRES_HOST, POSTGRES_PORT, and POSTGRES_DB.

–

Elasticsearch configuration

The following environment variables are written to ${SENTINEL_HOME}/etc/org.opennms.features.flows.persistence.elastic.cfg.

Variable Description Example value

ELASTICSEARCH_URL

Elasticsearch URL.

http://localhost:9200

ELASTICSEARCH_INDEX_STRATEGY

Index strategy.

hourly

ELASTICSEARCH_REPLICAS

Number of index replicas.

0

ELASTICSEARCH_CONN_TIMEOUT

Connection timeout in milliseconds.

30000

ELASTICSEARCH_READ_TIMEOUT

Read timeout in milliseconds.

60000

ELASTICSEARCH_USER

Username for Elasticsearch authentication (globalElasticUser).

elastic

ELASTICSEARCH_PASSWORD

Password for Elasticsearch authentication (globalElasticPassword).

–

Other ELASTICSEARCH_* variables are ignored, so the variables Kubernetes injects for a Service named elasticsearch (such as ELASTICSEARCH_PORT) do not end up in the configuration. To set any other property, overlay the configuration file (see Directory conventions).

Telemetry flow adapters

Variable Description Default value

SENTINEL_FLOWS_ENABLED

Set to true to install the sentinel-flows, sentinel-jsonstore-postgres, and sentinel-blobstore-noop features at startup. Must be true or false.

false

Individual adapters are defined by overlaying one configuration file per adapter into ${SENTINEL_HOME}/etc/ (see Directory conventions), named org.opennms.features.telemetry.adapters-<name>.cfg, where <name> uniquely identifies the adapter. This is the same factory-PID convention used to configure Minion listeners.

Example: org.opennms.features.telemetry.adapters-Netflow-5.cfg
name = Netflow-5
class-name = org.opennms.netmgt.telemetry.protocols.netflow.adapter.netflow5.Netflow5Adapter
parameters.some-key = some-value

Instance ID

Variable Description

OPENNMS_INSTANCE_ID

Sets org.opennms.instance.id in ${SENTINEL_HOME}/etc/custom.system.properties.

Prometheus JMX exporter

Variable Description Default value

PROM_JMX_EXPORTER_ENABLED

Set to true to enable the Prometheus JMX exporter.

false

PROM_JMX_EXPORTER_PORT

Port on which the JMX exporter listens.

9299

PROM_JMX_EXPORTER_CONFIG

Path to the exporter configuration file inside the container.

/opt/prom-jmx-exporter/config.yaml

PROM_JMX_EXPORTER_JAR

Path to the JMX Prometheus agent jar inside the container.

/opt/prom-jmx-exporter/jmx_prometheus_javaagent.jar
If you need a more sophisticated JMX configuration with custom rules, inject your own config.yaml file rather than relying on the default.

Directory conventions

Mountpoint Description

/opt/sentinel-etc-overlay

Files copied into ${SENTINEL_HOME}/etc/ at startup, after environment variable processing. Direct file overlays take precedence over the corresponding environment variable settings.

/opt/sentinel-overlay

Files copied into ${SENTINEL_HOME}/ at startup.

/keystore

Directory for the encrypted keystore file initialized with the -s flag.

Migrating from confd-based images

Earlier Sentinel images rendered configuration from a ${SENTINEL_HOME}/sentinel-config.yaml file with confd. This file is no longer supported, and the container refuses to start if it is present. To migrate, move each setting to its environment variable or overlay file, then remove the file:

sentinel-config.yaml key Replacement

location, id, broker-url

SENTINEL_LOCATION, SENTINEL_ID, OPENNMS_BROKER_URL

datasource.url

POSTGRES_JDBC_URL, or POSTGRES_HOST, POSTGRES_PORT, and POSTGRES_DB

datasource.username, datasource.password, datasource.database-name

POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB

ipc.kafka.*

KAFKA_IPC_* (see Kafka configuration); Kafka is selected automatically when KAFKA_IPC_BOOTSTRAP_SERVERS is set

elasticsearch.*

ELASTICSEARCH_*

telemetry.flows.adapters

SENTINEL_FLOWS_ENABLED=true and one adapter configuration file per adapter

org.opennms.instance.id

OPENNMS_INSTANCE_ID

java.agent.prom-jmx-exporter.*

A mounted config.yaml (see PROM_JMX_EXPORTER_CONFIG)

On startup, the container removes the files confd generated in existing volumes (etc/featuresBoot.d/ipc-strategy.boot and deploy/confd-flows-feature.xml).

Configuration files written by earlier images are kept when an existing etc volume is reused. In particular, org.opennms.sentinel.controller.cfg and org.opennms.netmgt.distributed.datasource.cfg keep the values written on their first start, so later changes to SENTINEL_LOCATION, OPENNMS_BROKER_URL, or the POSTGRES_* variables have no effect. To have them follow the environment, delete those two files from the volume; they are recreated on the next start, and you must set SENTINEL_ID to keep the existing ID.