SNMP Traps

Meridian can receive and process SNMP traps/informs from SNMP-capable devices out of the box. It receives SNMP traps via the trapd service daemon, which is enabled by default.

Meridian transforms these traps into events based on event definitions. Meridian includes trap definitions from many major manufacturers by default, and you can create your own event definitions.

Before you begin

There are a few tasks you must complete before your server can receive SNMP traps:

  1. Configure the port on which to receive SNMP traps (see Receive SNMP traps/informs). By default, Meridian listens on a non-standard port for trap messages.

  2. Ensure that SNMP-capable devices on your network are configured to send traps to Meridian.

SNMP trap definition

You can configure SNMP traps to customize the types of information you receive and the conditions under which Meridian converts traps into an event. For example, you may want to see an event only when a device is down, or when a combination of criteria are met (for example, OID X with "major" severity level and status "indeterminate"). These traps are transformed into events only when the conditions you define are met.

There are several ways to configure Meridian to customize the events you receive and see:

  • Determine what problem you are trying to solve (for example, "I want to know X about my Cisco router").

  • Manually send the corresponding event to the Meridian server and see how the event is handled (see Manually send events).

  • Import event definitions with the SNMP MIB compiler.

  • Configure the event to meet your requirements, if required.

Create event definition for an SNMP trap

Meridian includes event definitions for traps from many vendors' equipment. Use the mask tag to match SNMP traps in eventconf.xml. See Anatomy of an event for details on event structure.

The following example shows a Cisco Systems event for their C3800 device. Parts of it look similar to internally generated events, with the main difference being the mask block. This block consists of <maskelement> tags, and the event will match only if all of the defined tags are met.

Cisco systems event
<event>
  <mask>
    <maskelement>
      <mename>id</mename>
      <mevalue>.1.3.6.1.4.1.9.9.70.2</mevalue>
    </maskelement>
    <maskelement>
      <mename>generic</mename>
      <mevalue>6</mevalue>
    </maskelement>
    <maskelement>
      <mename>specific</mename>
      <mevalue>17</mevalue>
    </maskelement>
  </mask>
  <uei>http://uei.opennms.org/vendor/Cisco/traps/ciscoC3800SysAggregateStatusChange</uei>
  <event-label>CISCO-C3800-MIB defined trap event: ciscoC3800SysAggregateStatusChange</event-label>
  <descr>&#38lt;p&#38gt;Notification that the aggregate status of a node
         has changed.&#38lt;/p&#38gt;&#38lt;table&#38gt;
         &#38lt;tr&#38gt;&#38lt;td&#38gt;&#38lt;b&#38gt;
         c3800SysNextTrapSeqNum&#38lt;/b&#38gt;</td&#38gt;&#38lt;td&#38gt;%parm[#1]%
         &#38lt;/td&#38gt;&#38lt;td&#38gt;&#38lt;p;&#38gt;</p&#38gt;&#38lt;/td;&#38gt;&#38lt;/tr&#38gt;&#38lt;tr&#38gt;&#38lt;td&#38gt;&#38lt;b&#38gt;
         sysName&#38lt;/b&#38gt;&#38lt;/td&#38gt;&#38lt;td&#38gt;%parm[#2]%
         &#38lt;/td&#38gt;&#38lt;td&#38gt;&#38lt;p;&#38gt;&#38lt;/p&#38gt;&#38lt;/td;&#38gt;&#38lt;/tr&#38gt;&#38lt;tr&#38gt;&#38lt;td&#38gt;&#38lt;b&#38gt;
         c3800SysTrapSeverity&#38lt;/b&#38gt;&#38lt;/td&#38gt;&#38lt;td&#38gt;%parm[#3]%
         &#38lt;/td&#38gt;&#38lt;td&#38gt;&#38lt;p;&#38gt;
         clear(1) minor(2) major(3)&#38lt;/p&#38gt;
         &#38lt;/td;&#38gt;&#38lt;/tr&#38gt;&#38lt;tr&#38gt;&#38lt;td&#38gt;&#38lt;b&#38gt;
         c3800SysAggregateStatus&#38lt;/b&#38gt;&#38lt;/td&#38gt;&#38lt;td&#38gt;%parm[#4]%
         &#38lt;/td&#38gt;&#38lt;td&#38gt;&#38lt;p;&#38gt;
         clear(1) minor(2) major(3)</p>
         &#38lt;/td;&#38gt;&#38lt;/tr&#38gt;&#38lt;/table&#38gt;
  </descr>
  <logmsg dest='logndisplay'><p>Cisco Event: C3900: Node Status has changed.</p></logmsg>
  <severity>Indeterminate</severity>
</event>

This event will match an SNMP trap where the enterprise OID (id) is equal to .1.3.6.1.4.1.9.9.70.2, the generic trap value is equal to 6 (enterprise-specific), and the vendor trap value is equal to 17.

Possible <mename> values include the following:

  • uei

  • source

  • host

  • snmphost

  • nodeid

  • interface

  • service

  • id

  • specific (vendor-specific error conditions and error codes. Refer to the associated MIB for details.)

  • generic (fixed, universal SNMP trap types)

    • Coldstart or warmstart

    • Linkup or linkdown

    • Authentication fails

    • egpNeighborloss (Agent cannot communicate with its EGP (Exterior Gateway Protocol) peer.)

  • community (community string in an SNMP trap)

Note that the following elements forward and filters are deprecated.

You can use the % symbol as a wildcard in the mask values. For example, to match all Cisco events, you could use the following:

<mask>
  <maskelement>
    <mename>id</mename>
    <mevalue>.1.3.6.1.4.1.9.%</mevalue>
  </maskelement>
</mask>

The order in which events are listed in the eventconf.xml file is extremely important. The search stops with the first event definition that matches the given event (see eventconf.xml). As such, if the code with the wildcard is listed before the more specific ciscoC3800SysAggregateStatusChange event, the latter event will never be generated.

Also note that the wildcard is simply a substring match. If a device generates an event with the enterprise OID of .1.3.6.1.4.1.9 it would not match this event, as there is no trailing ".". If the trailing "." is left off, you must take care so that a trap with an OID of .1.3.6.1.4.1.99 is listed before the .1.3.6.1.4.1.9% event or else it will match the more generic event.

Use the parm replacement token with trap events

Some events, especially SNMP traps, have additional information sent with them called "variable bindings" (varbinds). In the ciscoC3800SysAggregateStatusChange event listed above, there are four of them. You can use the parm replacement token to access them. Each parameter consists of a name and a value.

For example, the ciscoC3800SysAggregateStatusChange event description lists out each of the parameters. Thus the second parameter, the sysName, is printed using %parm[#2]%.

See Parameter tokens for more information.

Filter on varbinds

Variable bindings (varbinds) are key-value pairs that provide alert data in SNMP traps. You can use varbinds as filters for further granularity on the conditions for creating an event through the mask block.

Using our previous ciscoC3800SysAggregateStatusChange example, what should its severity be? The event is generated whenever the status changes, but we don’t know if the change is "bad" (from operational to non-operational) or "good" (the non-operational status is cleared).

The parameters passed with the event contain that information, particularly parameter #3, the trap severity.

We rewrite our event as follows:

<mask>
  <maskelement>
    <mename>id</mename>
    <mevalue>.1.3.6.1.4.1.9.9.70.2</mevalue>
  </maskelement>
  <maskelement>
    <mename>generic</mename>
    <mevalue>6</mevalue>
  </maskelement>
  <maskelement>
    <mename>specific</mename>
    <mevalue>17</mevalue>
  </maskelement>
  <varbind>
    <vbnumber>3</vbnumber>
    <vbvalue>3</vbvalue>
  </varbind>
</mask>

In the example, adding a mask with a varbind tag will match on the same ID, generic, and specific values, but also requires that the third parameter is equal to "3" (indicating a Cisco-determined trap severity of "major").

With a "status change" event, you may want to create separate events for each status value. To do this, copy the event definition once for each status value, add the varbind mask, and then change the UEI, description, severity, and logmsg to match the event.

You can also match more than one varbind and more than one value per varbind:

<varbind>
  <vbnumber>3</vbnumber>
  <vbvalue>2</vbvalue>
  <vbvalue>3</vbvalue>
</varbind>
<varbind>
  <vbnumber>4</vbnumber>
  <vbvalue>2</vbvalue>
  <vbvalue>3</vbvalue>
</varbind>

The above code snippet will match if the third parameter has a value of "2" or "3" and the fourth parameter has a value of "2" or "3".

You can also use a regular expression match on the varbind value. Just specify the expression prefixed with a with a "~":

<varbind>
  <vbnumber>1</vbnumber>
  <vbvalue>~[Dd]own</vbvalue>
</varbind>

This will match a varbind 1 that contains the word "Down" or "down" anywhere within its value. You can also do quick prefix matches with the '%' in a varbind value:

<varbind>
  <vbnumber>1</vbnumber>
  <vbvalue>Error:%</vbvalue>
</varbind>

This will match varbind 1 with any String beginning with "Error:".

REMINDER: The order in which events are listed is very important. Put the most specific events first.

Decode varbinds

A lot of MIBs define specific variables to code the value of some OIDs. As an example, the SNMP agent returns a numerical value for the ifAdminStatus and ifOperStatus: 1 means Up and 2 means Down.

Because Meridian does not have a MIB parser, we usually put this map (between the numerical encoded value and its meaning) into the event description. Use the varbindsdecode element to convert the event varbind value into a decoded String.

With varbindsdecode, Meridian decodes the numerical value sent into trap varbinds to the corresponding String values, which then can be substituted into the logmsg.

For example, a Cisco HSRP status change trap (OID `.1.3.6.1.4.1.9.9.106.2, generic 6, and specific 1), which corresponds to the uei.opennms.org/vendor/Cisco/traps/cHsrpStateChange event.

The trap contains the following varbind: cHsrpGrpStandbyState, with possible values from 1 to 6:

  • initial(1)

  • learn(2)

  • listen(3)

  • speak(4)

  • standby(5)

  • active(6)

The following is the original event definition:

<event>
 <mask>
  <maskelement>
   <mename>id</mename>
   <mevalue>.1.3.6.1.4.1.9.9.106.2</mevalue>
  </maskelement>
  <maskelement>
   <mename>generic</mename>
   <mevalue>6</mevalue>
  </maskelement>
  <maskelement>
   <mename>specific</mename>
   <mevalue>1</mevalue>
  </maskelement>
 </mask>
 <uei>uei.opennms.org/vendor/Cisco/traps/cHsrpStateChange</uei>
 <event-label>CISCO-HSRP-MIB defined trap event: cHsrpStateChange</event-label>
 <descr><p>A cHsrpStateChange notification is sent when a cHsrpGrpStandbyState transitions to either active or standby state, or leaves active or standby state. There will be only one notification issued when the state change
 is from standby to active and vice versa.</p><table>
 <tr><td><b>
 cHsrpGrpStandbyState</b></td><td>%parm[#1]%
 </td><td><p;>
 initial(1) learn(2) listen(3) speak(4) standby(5) active(6)</p>
 </td;></tr></table>
 </descr>
 <logmsg dest='logndisplay'><p>Cisco Event: HSRP State Change.</p></logmsg>
 <severity>Minor</severity>
 </event>

The following example shows how to change the event definition so that the HSRP status is decoded and the literal definition is displayed inside the logmsg:

<event>
 <mask>
  <maskelement>
   <mename>id</mename>
   <mevalue>.1.3.6.1.4.1.9.9.106.2</mevalue>
  </maskelement>
  <maskelement>
   <mename>generic</mename>
   <mevalue>6</mevalue>
  </maskelement>
  <maskelement>
   <mename>specific</mename>
   <mevalue>1</mevalue>
  </maskelement>
 </mask>
 <uei>uei.opennms.org/vendor/Cisco/traps/cHsrpStateChange</uei>
 <event-label>CISCO-HSRP-MIB defined trap event: cHsrpStateChange</event-label>
 <descr><p>A cHsrpStateChange notification is sent when a cHsrpGrpStandbyState transitions to either active or standby state, or leaves active or standby state. There will be only one notification issued when the state change
 is from standby to active and vice versa.</p><table>
 <tr><td><b>
 cHsrpGrpStandbyState</b></td><td>%parm[#1]%
 </td><td><p;>
 initial(1) learn(2) listen(3) speak(4) standby(5) active(6)</p>
 </td;></tr></table>
 </descr>
 <logmsg dest='logndisplay'><p>Cisco Event: HSRP State Change to %parm[#1]%.</p></logmsg>
 <severity>Minor</severity>
 <varbindsdecode>
 <parmid>parm[#1]</parmid>
 <decode varbindvalue="1" varbinddecodedstring="initial"/>
 <decode varbindvalue="2" varbinddecodedstring="learn"/>
 <decode varbindvalue="3" varbinddecodedstring="listen"/>
 <decode varbindvalue="4" varbinddecodedstring="speak"/>
 <decode varbindvalue="5" varbinddecodedstring="standby"/>
 <decode varbindvalue="6" varbinddecodedstring="active"/>
 </varbindsdecode>
</event>

The first varbind into the trap (parm[#1]) is translated using the decode map. If the value of the first OID in this trap is 6, the the log message will be the following:

<p>Cisco Event: HSRP State Change to active.</p>

Trap value representation

When octet Strings are translated into event parameters, Meridian first attempts to treat them as character encodings. If all bytes in the String are valid UTF-8 or ISO-8859-1 characters, the String is stored as these characters. If this is not possible, the value is encoded as a Base64 String.

Traps forwarded via proxy

When SNMP traps are forwarded through a proxy using SNMPv2c or SNMPv3, preserving the original source IP address is a challenge due to the lack of an agent-addr field in the TRAP-V2 PDU used in those protocol versions. RFC 3584 defines an optional varbind snmpTrapAddress (.1.3.6.1.6.3.18.1.3.0) that you can add to forwarded traps to convey the original source IP address.

To configure Meridian to honor snmpTrapAddress when present, set use-address-from-varbind="true" in the top-level element of ${OPENNMS_HOME}/etc/trapd-configuration.xml and restart Meridian.

Configuration example for using RFC 3584 helper varbinds in forwarded traps
<trapd-configuration snmp-trap-port="1162"(1)
                     new-suspect-on-trap="false"(2)
                     use-address-from-varbind="true" />(3)
1 Set the SNMP trap daemon listening port to 1162/udp.
2 Don’t create new nodes when receiving an SNMP trap with an unknown source IP address.
3 Try to use the identifier source IP address from the snmpTrapAddress varbind instead of the UDP source IP address.

If you are using Minions to receive traps, edit the ${MINION_HOME}/etc/org.opennms.netmgt.trapd.cfg file on each Minion to include the following setting:

trapd.useAddressFromVarbind=true