Flows Troubleshooting

This section describes ways to troubleshoot flows when the feature does not work or performs suboptimally.

Troubleshoot telemetryd

Telemetryd receives and decodes flows on Horizon, so you can run the following checks to determine that it works as expected:

  • Telemetryd availability

  • Are routers sending data?

  • Docker networking

  • Minion health check

  • OpenNMS health check

  • Are sink consumer graphs populated?

  • Is SNMP available on the routers that provide Netflow?

  • Review OpenNMS and Minion logs

The Troubleshoot Telemetryd article on Discourse provides details on how to run these checks.

No data or incorrect data

In a scenario where you see no data or incorrect data, you should view the state and parameters of telemetry listeners, and whether they are processing data and information with the following command: opennms:telemetry-listeners

Check Elasticsearch persistence

If you store flows in Elasticsearch, you can use Kibana to check if flow documents (raw and/or aggregated) are written to Elasticsearch. You must know your endpoint address and API key.

Run the following curl command:

curl "https://your-es-server.hostname:9200/_cat/indices?v&apikey=Your-API-Key"

The query returns a list of indices. Those that start with a . are system indices. All others are regular indices. Regular indices appear only when Elasticsearch is receiving flows.

No Elasticsearch persistence

If you do not see regular indices, in OpenNMS check etc/telemetryd-configuration.xml. You must enable Netflow listeners.

Persisted flows do not appear in Helm

If you have persisted flows but they do not appear in Helm, check the configuration in etc/org.opennms.features.flows.persistence.elastic.cfg, in particular, the elasticUrl property. If you are using aggregated flows, make sure your aggregate.elasticIndexStrategy matches the index strategy you configured in the streaming analytics tool. To persist only raw flows or only aggregated flows in Elasticsearch, you must set alwaysUseRawForQueries or alwaysUseAggForQueries appropriately.

(For more information on troubleshooting Elasticsearch, refer to the Elasticsearch documentation.)

Verify flows by device

In the UI, use the Nodes page to determine flows performance for specific devices. (Choose Info>Nodes to view nodes.) The flows indicator icon shows flows data for each device, with SNMP details and flows direction.

Replay flows from a packet capture

To help debug flow processing, use the following Karaf shell commands to replay flows from a packet capture:

List the available listeners and parsers:

opennms:telemetry-listeners

Replay the packet capture (.pcap) to the target parser from the output above:

opennms:telemetry-replay-pcap <listener> <parser> <path-to-pcap-file>

Here’s an example that replays a .pcap with Netflow 9 flows to the Netflow 9 parser:

admin@opennms()> opennms:telemetry-listeners
Name = Multi-UDP-9999
Description = UDP *:9999
Properties:
  Max Packet Size = 8096
  Port = 9999
Parsers:
  - Multi-UDP-9999.Netflow-5-Parser
  - Multi-UDP-9999.Netflow-9-Parser
  - Multi-UDP-9999.IPFIX-TCP-Parser
  - Multi-UDP-9999.SFlow-Parser
admin@opennms()> opennms:telemetry-replay-pcap Multi-UDP-9999 Multi-UDP-9999.Netflow-9-Parser /tmp/flows.pcap
Processing packets from '/tmp/flows.pcap'.
Processing packet #100.
Processing packet #200.
Processing packet #300.
Processing packet #400
Processing packet #500.
Done processing 515 packets.
admin@opennms()>

Flows will be ingested using the same ingest and processing pipeline as they would if received directly from the devices. Nodes with interfaces that match the IP addresses in the .pcap must exist in order to associate the results with a node.