The telemetry daemon (telemetryd) provides an extensible framework that you can use to handle sensor data pushed to Meridian. You can use the framework to support applications that use different protocols to transfer metrics data.
With telemetryd, operators can define listeners that support different protocols to receive the telemetry data, and adapters that transfer the received data into generic formats like flows or performance data.
The configuration is split in two parts: on one side, listeners and their attached parsers or connectors receive telemetry data that is transported over a specific protocol, and parse the data according to the protocol’s specification. On the other side, adapters pick up the parsed data and enrich it before persisting it.
Queues transport the parsed telemetry data from parsers to adapters by binding a specific parser to a specific adapter.
Listeners receive sensor data from an external source and hand the data off to parsers, which handle the transport protocol format and extract the transported data. Listeners do the transport protocol handling, like opening sockets and accepting incoming connections. The received messages are then passed to the parsers that are configured to the listeners. They parse the concrete protocol implementation and extract all available information from the messages. The extracted data is then handed over to the configured queue for further processing.
For example, a listener may open a UDP socket to receive packets, and an assigned parser parses the incoming IPFIX packets.
Parsers and listeners must work closely together; for that reason, they cannot be combined randomly. The supported parsers are described in the documentation for each listener. For more information on listeners, their supported parsers, and how to configure them, see Telemetryd listeners reference.
Connectors initiate connections and maintain sessions with agents. A new connector is created for every agent that it is responsible for connecting to. Here, "agent" refers to a named IP service attached to an IP interface on a specific node.
Connectors do not use parsers; they dispatch messages directly to queues.
Adapters enrich and process data that is dispatched over a queue. They receive the data from the queue and transform it into a format that can be used by Meridian. They can also enrich incoming data to attach node or interface information, depending on the information that is already available. For example, an adapter may be used to transform telemetry data that is received via sFlow into performance data and persist it to the applicable node.
Since adapters must handle data dependent on a transport protocol, not all of them can be used in combination with every possible parser. The supported parsers are described in the documentation for each adapter.
If you have multiple adapters, the order in which they are run is the same as defined in
Queues transfer data between parsers and adapters, and are represented by a channel in the messaging system.
The following options can help fine-tune queue behavior:
|Queue Attribute (Meridian)
Number of threads used for consuming and dispatching messages.
(2 * number of cores)
Maximum number of messages to keep in memory while waiting to be dispatched.
Whether to use the routing key when forwarding messages to the broker. This enforces message ordering.
Maximum batch size. When the batch reaches this size, it is dispatched.
Maximum batch interval, in milliseconds. When the batch has been created for longer than this interval, it is dispatched, regardless of the current size.
When using Kafka as a message broker, setting
false balances messages across all partitions.
This can be done safely for flows, but is not supported for metrics when using thresholding (order is required).
In Meridian, you can add these options to the
<queue> element as attributes:
<queue name="IPFIX" use-routing-key="false">
On a Minion, you can add these options as parser properties, and as adapter properties on Sentinel:
Listeners and their parsers may run either on Meridian or a Minion, whereas adapters may run either on Meridian or Sentinel. If a listener and its parsers are running on a Minion, the received messages are dispatched automatically via a queue to associated adapters running in Meridian or Sentinel. Minions can receive data, but require Meridian or Sentinel to process it.
Some parsers support reverse hostname resolution to annotate IP addresses using the actual hostname. The Minion performs the reverse lookup while it parses the protocol data. The resolved hostname, if found, is attached to the address information. Both are then sent to the adapter for further processing. For more information see DNS Resolver.
The following parsers currently support reverse hostname resolution: