Flows Troubleshooting

You can run the following checks to determine that telemetryd is working as expected:

The Troubleshoot Telemetryd Discourse article describes how to run these checks.

In a scenario where you see no data or incorrect data, you should view the state and parameters of telemetry listeners:

If you store flows in Elasticsearch, you can use Kibana to check if flow documents (raw and aggregated) are written to Elasticsearch.

To check Elasticsearch persistence, run this command:

The query returns a list of indices. Those that start with a . are system indices; all others are regular indices. Regular indices appear only when Elasticsearch is receiving flows. If no regular indices are displayed, check etc/telemetryd-configuration.xml and verify that Netflow listeners are enabled.

For more information on troubleshooting Elasticsearch, refer to the official Elasticsearch documentation.

If your persisted flows are not displayed in the OpenNMS plugin for Grafana, check the elasticUrl property in etc/org.opennms.features.flows.persistence.elastic.cfg. If you are using aggregated flows, ensure that your aggregate.elasticIndexStrategy matches the index strategy that you configured in the streaming analytics tool. To persist only raw flows or only aggregated flows, set either alwaysUseRawForQueries or alwaysUseAggForQueries, as appropriate.

To verify flows by device, click Info  Nodes in the web UI’s top menu bar. The flows indicator symbol on the Nodes page shows flows data for each device, with SNMP details and the direction of flows.

You can replay flows from a package capture to help debug flow processing:

This example is the result of replaying a package capture with Netflow 9 flows to the Netflow 9 parser:

Flows are ingested using the same pipeline that they would if received directly from the devices. Nodes with interfaces that match the IP addresses in the package capture must exist to associate the results with a node.

Flow analyses use timestamps that are exposed by the underlying flow management protocol. The timestamps are set depending on the exporting router’s clock settings. If the router’s clock differs from the actual time, the skew is reflected in the received flows; this can subsequently impact further analysis and aggregation.

Horizon can correct skewed flow timestamps. To do so, it compares the exporting device’s current time with the actual time that the packet is received. If the times differ significantly, the time that the flow was received is considered more correct and all timestamps associated with the flow are updated.

To enable clock correction, configure a threshold for the maximum allowed delta, in milliseconds. Note that setting the threshold to 0 disables the correction mechanism.