Trouble Ticket Notifications

Horizon normally manages trouble tickets from alarms, either manually from the web UI or through the automations described in Ticketing. The ticket notification strategy provides a third path: it creates a ticket for the alarm associated with a notification’s event, and closes that ticket once the alarm clears. Because it runs through notifd, it can take advantage of notification features that alarm automations do not, such as destination paths, escalation, and path outage suppression.

Notifd delivers the same command for resolutions as well as problems, so the strategy reads the alarm behind the event rather than the event itself:

How the strategy acts on the alarm behind the notification’s event
Alarm	Ticket	Action
Not cleared	None, or closed, resolved, cancelled, or creation failed	Create a ticket
Not cleared	Active, or a create already in flight	Nothing
Cleared	Active	Close the ticket
Cleared	None, or already settled	Nothing

Requirements and caveats

A ticketer plugin must be configured. Without one, the ticket events that this strategy sends are not acted on.
The event that triggers the notification must have alarm-data in its event definition. Events without alarm data are skipped, and the notification completes without creating a ticket.
The strategy looks up the alarm for the event at delivery time, but alarmd performs that association asynchronously. A destination path that fires immediately can run before the alarm exists, in which case the notification fails with a "no alarm-id" error. Use a destination path with a non-zero initial-delay to avoid this; 30 seconds is usually enough.
The destination path target’s contact information is not used, whether the target is a user or a group. The strategy reads only event parameters, so the target serves to satisfy the destination path schema.
Target a single user rather than a group. Notifd runs each command once per member of a group target, and each run sends its own create-ticket event. The strategy skips the create event when the alarm already has an active ticket; tickets that are closed, resolved, or cancelled, or whose creation failed, do not block a new one. Near-simultaneous deliveries can race the asynchronous ticket creation, so a group target can still produce duplicate tickets.
The ticket events carry a user parameter that defaults to admin. You can override it with the optional ticketUser argument described below. Whether the value appears on the created ticket depends on the ticketer plugin.
The ticketer ignores close requests for alarms that are not cleared, controlled by opennms.ticketer.skipCloseWhenNotCleared (default true). The strategy applies the same condition, so it sends a close only once the alarm is cleared.
Closing a ticket leaves its id on the alarm with the state Closed. A later occurrence of the same alarm opens a new ticket rather than reopening the closed one.

Setup

Add the following command to notificationCommands.xml:

<command binary="false">
    <name>troubleTicket</name>
    <execute>org.opennms.netmgt.notifd.TicketNotificationStrategy</execute>
    <comment>class for managing the trouble ticket for the alarm associated with the notification's event</comment>
    <argument streamed="false">
        <switch>eventID</switch>
    </argument>
    <argument streamed="false">
        <switch>eventUEI</switch>
    </argument>
    <argument streamed="false">
        <switch>noticeid</switch>
    </argument>
    <argument streamed="false">
        <switch>ticketUser</switch>
    </argument>
</command>

Add destination paths with an initial delay to destinationPaths.xml. Both run the same command, one for each direction:

<path name="Create-Ticket" initial-delay="30s">
    <target>
        <name>admin</name>
        <command>troubleTicket</command>
    </target>
</path>
<path name="Close-Ticket" initial-delay="30s">
    <target>
        <name>admin</name>
        <command>troubleTicket</command>
    </target>
</path>

Use a single user as the target, not a group. Notifd runs the command once per member of a group target, and near-simultaneous runs can each create a ticket for the same alarm.

Reference the Create-Ticket destination path from each notification in notifications.xml that should open a ticket, and Close-Ticket from the notifications for the events that resolve them.

To set the ticket user for a notification, add a ticketUser parameter to its definition:

<parameter name="ticketUser" value="noc"/>

If the parameter is absent, the strategy uses admin.

Closing tickets

Add a notification for the event that resolves the problem and point it at the Close-Ticket path:

<notification name="nodeUp-ticket" status="on">
    <uei>uei.opennms.org/nodes/nodeUp</uei>
    <rule>IPADDR != '0.0.0.0'</rule>
    <destinationPath>Close-Ticket</destinationPath>
    <text-message>Closing the trouble ticket for node %nodelabel%.</text-message>
    <parameter name="ticketUser" value="noc"/>
</notification>

The resolving event is reduced onto the same alarm as the problem event, so the strategy finds that alarm’s ticket and closes it. The initial-delay on the path is what makes this reliable: alarmd clears the alarm asynchronously, and the ticketer refuses to close a ticket whose alarm is not yet cleared.

Notifd may also deliver this command a second time for the same resolution, because acknowledging the problem notice re-runs its commands. That delivery finds the ticket already closed and does nothing.