Thresholding Service
The Thresholding Service maintains the state of the performance metrics and generates alarms when thresholds are triggered (armed) or cleared (unarmed). The service listens for and compares performance metrics after they are persisted to the time series database. The state of the thresholds are held in memory and pushed to persistent storage only when they change.
Distributed thresholding with Sentinel
Thresholding for streaming telemetry with telemetryd is supported on Sentinel when using Newts. When running on Sentinel, the thresholding state can be stored in either Cassandra or PostgreSQL. Given that Newts requires Cassandra, we recommend using Cassandra to minimize the load on PostgreSQL.
Thresholding on Sentinel uses the same configuration files as Horizon and operates similarly.
When a thresholding changes to or from trigger or cleared, an event is published which is processed by Horizon and the alarm is created or updated.