Basic Flows Setup

This section describes how to get started with flows to collect, enrich (classify), persist, and visualize flows.

Requirements

Make sure that you have the following before you set up flows:

  • A configured Horizon instance.

  • One or more devices that send flows visible to Horizon, and that are monitored with SNMP.

  • Elasticsearch cluster set up with the Elasticsearch Drift plugin installed on every Elasticsearch node.

    • The Drift plugin persists and queries flows that Horizon collects. The Drift version must match the targeted Elasticsearch version.

    • (Optional) Configure Elasticsearch variables like search.max_buckets or maximum heap size ES-JAVA_OPTS if the default values are not sufficient for your volume of flows or number of nodes.

    • (Optional) Create a job to clean the indices so that the disk does not fill up (for example, "keep X days of flows"). Filled disks are a challenging problem to address for those who are not Elasticsearch experts. We recommend using the Elasticsearch Curator tool to do this.

    • Horizon set up to monitor the Elasticsearch stack and generate an alarm if it goes down.

  • The OpenNMS plugin for Grafana configured to visualize flows.

    • Configure the flow and performance data sources.

Configure Horizon to communicate with Elasticsearch

Horizon must be set up to communicate with Elasticsearch and persist the collected flows data in a defined directory:

  1. Connect to your Horizon Karaf shell:

    ssh -p 8101 admin@localhost

  2. Edit ${OPENNMS_HOME}/etc/org.opennms.features.flows.persistence.elastic.cfg to configure flow persistence to use your Elasticsearch cluster:

    config:edit org.opennms.features.flows.persistence.elastic
config:property-set elasticUrl http://elastic:9200
config:update

  3. (Optional) Edit or create ${OPENNMS_HOME}/etc/org.opennms.features.flows.persistence.elastic.cfg and configure persistence settings:

    elasticUrl = http://10.10.3.218:9200 (1)
connTimeout = 30000
readTimeout = 300000
settings.index.number_of_replicas = 0
settings.index.number_of_shards=1
settings.index.refresh_interval=10s
elasticIndexStrategy=daily
    1 Replace with comma-separated list of Elasticsearch nodes.

Multi-protocol listener

With most tools, if you are monitoring multiple flow protocols, you must set up a listener on its own UDP port for each protocol. However, Horizon allows a multi-port listener option; this listener, named Multi-UDP-9999, is enabled by default and monitors multiple protocols on a single UDP port (9999). The default configuration includes support for Netflow v5, Netflow v9, sFlow, and IPFIX. You can edit ${OPENNMS_HOME}/etc/telemetryd-configuration.xml to change the port number and add or remove protocols.

Make sure your firewall allow list includes the ports that you have configured to receive flow data.

Enable individual protocols

Follow these steps to enable one or more protocols that you want to handle individually or that are not included in the default multi-protocol listener.

This example uses the NetFlow v5 protocol. You can follow the same steps for any of the other flow-related protocols.

  1. Edit ${OPENNMS_HOME}/etc/telemetryd-configuration.xml to enable protocols:

    <listener name="Netflow-5-UDP-8877" class-name="org.opennms.netmgt.telemetry.listeners.UdpListener" enabled="true">
    <parameter key="port" value="8877"/>

    <parser name="Netflow-5-Parser" class-name="org.opennms.netmgt.telemetry.protocols.netflow.parser.Netflow5UdpParser" queue="Netflow-5" />
</listener>

<queue name="Netflow-5">
    <adapter name="Netflow-5-Adapter" class-name="org.opennms.netmgt.telemetry.protocols.netflow.adapter.netflow5.Netflow5Adapter" enabled="true">
    </adapter>
</queue>
    The default configuration file provides example configurations for many protocols. To enable one of these protocols, find the correct example listener and adapter elements and change their enabled attributes to true.

  2. Reload the daemon to apply your changes:

    ${OPENNMS_HOME}/bin/send-event.pl -p 'daemonName Telemetryd' uei.opennms.org/internal/reloadDaemonConfig

This configuration opens a UDP socket bound to 0.0.0.0:8877 to listen and process NetFlow v5 messages that are are forwarded to this port.

Enable flows on your devices

Refer to your device’s manufacturer documentation to learn more about configuring it to send flows.

You may need to set up the flow receiver (in this case, Horizon) and allow sending flows per interface on your firewall.

You must configure a connection to your instance of the OpenNMS plugin for Grafana to access flow-related graphs from the Horizon web UI:

  1. Connect to your Horizon Karaf shell:

    ssh -p 8101 admin@localhost

  2. Configure the plugin settings:

    config:edit org.opennms.netmgt.flows.rest
config:property-set flowGraphUrl 'http://grafana:3000/dashboard/flows?node=$nodeId&interface=$ifIndex' (1)
config:update
    1 This URL can also point to other tools. It supports the $nodeId, $ifIndex, $start, and $end placeholders.

After the plugin is configured, an icon is displayed at the top-right corner of an SNMP resource graph indicating that flow data is available for the interface. If you have trouble during or after configuration, refer to Flows Troubleshooting.

Configure Kafka forwarder

Flows enriched with Horizon node data can be forwarded to Kafka and persisted. By default, enriched flows are stored in the flowsDocument topic and the payloads are encoded using Google Protocol Buffers. See flowdocument.proto in the corresponding source definition for the model definitions.

To enable JSON support, set useJson to true.

Follow these steps to configure forwarding flows to Kafka:

  1. Enable Kafka forwarding:

    $ ssh -p 8101 admin@localhost
...
admin@opennms()> config:edit org.opennms.features.flows.persistence.elastic
admin@opennms()> config:property-set enableForwarding true
admin@opennms()> config:update

  2. Configure the Kafka server for flows:

    $ ssh -p 8101 admin@localhost
...
admin@opennms()> config:edit org.opennms.features.flows.persistence.kafka
admin@opennms()> config:property-set bootstrap.servers 127.0.0.1:9092
admin@opennms()> config:property-set topic opennms-flows
admin@opennms()> config:update

Next steps

After you set up basic flows monitoring, you may want to do some of the following tasks: