XmlCollector

The XmlCollector collects and extracts metrics from XML and JSON documents.

Collector facts

Class Name

org.opennms.protocols.xml.collector.XmlCollector

Package

core

Supported on Minion

Yes (see limitations)

Configuration Files

${OPENNMS_HOME}/etc/xml-datacollection-config.xml
${OPENNMS_HOME}/etc/xml-datacollection.d/*.xml

Configuration and use

Table 1. Collector-specific parameters for the XmlCollector
Parameter Description Default

Required

collection

The name of the XML Collection to use.

n/a

Optional

handler-class

Class that performs the collection.

org.opennms.protocols.xml.collector.DefaultXmlCollectionHandler

The available handlers include:

  • org.opennms.protocols.xml.collector.DefaultXmlCollectionHandler

  • org.opennms.protocols.xml.collector.Sftp3gppXmlCollectionHandler

  • org.opennms.protocols.xml.vtdxml.DefaultVTDXmlCollectionHandler

  • org.opennms.protocols.xml.vtdxml.Sftp3gppVTDXmlCollectionHandler

  • org.opennms.protocols.json.collector.DefaultJsonCollectionHandler

  • org.opennms.protocols.http.collector.HttpCollectionHandler

Caveats

The org.opennms.protocols.json.collector.DefaultJsonCollectionHandler requires the fetched document to be a single element of type object to make xpath query work. If the root element is an array, it will be wrapped in an object accessible as /elements.

JSON object keys are not restricted the way XML element names are: a key may start with a digit or contain characters that are not allowed in an XML name, for example 2xx or o’brien. A plain XPath location step cannot express such a name, so the JSON collection handler rewrites those steps transparently to an equivalent *[name()='…​'] step before evaluating them. You can therefore write the key exactly as it appears in the JSON document:

<xml-group name="http-stats" resource-type="node" resource-xpath="/stats">
  <xml-object name="ok" type="GAUGE" xpath="2xx/count"/>
  <xml-object name="failed" type="GAUGE" xpath="5xx/count"/>
</xml-group>

This works for resource-xpath, key-xpath, timestamp-xpath and the xpath of an xml-object, and predicates keep working on a rewritten step, for example 2xx[1]/count. A key that holds an array still produces one resource per array element.

The rewrite applies only to the JSON collection handler and only to steps that are not already a legal XML name. It leaves predicates, functions, axis specifiers (self::, child::), ., .., *, node() and text() untouched, which has two consequences worth knowing:

  • A step written with an explicit axis or namespace prefix is not rewritten — write 2xx rather than child::2xx.

  • A value that is a full XPath expression rather than a location path, such as a union like /a | /b, is left as-is.

XML collection configuration

Understanding resource types helps when editing collector-specific configuration files.

XML collections are defined in ${OPENNMS_HOME}/etc/xml-datacollection-config.xml and groups are defined in ${OPENNMS_HOME}/etc/xml-datacollection/.

Sample collection definition named xml-opennms-nodes
<xml-collection name="xml-opennms-nodes">
  <rrd step="300">
    <rra>RRA:AVERAGE:0.5:1:2016</rra>
    <rra>RRA:AVERAGE:0.5:12:1488</rra>
    <rra>RRA:AVERAGE:0.5:288:366</rra>
    <rra>RRA:MAX:0.5:288:366</rra>
    <rra>RRA:MIN:0.5:288:366</rra>
  </rrd>
  <xml-source url="http://admin:admin@{ipaddr}:8980/opennms/rest/nodes">
    <import-groups>xml-datacollection/opennms-nodes.xml</import-groups>
  </xml-source>
</xml-collection>

The url attribute of the <xml-source> supports node and IpInterface field placeholders.

Metadata expressions can also be used in attributes of the xml-datacollection-config.xml configuration file. This lets the user to also reference credentials stored in the secure credentials.
The referenced opennms-nodes.xml file
<xml-groups>
    <xml-group name="nodes" resource-type="node" resource-xpath="/nodes">
        <xml-object name="totalCount" type="GAUGE" xpath="@totalCount"/>
    </xml-group>
</xml-groups>

Customize the source request

If you need to customize the request to pull raw data, you can add a <request> element within the <xml-source> block.

  <xml-source url="http://admin:admin@{ipaddr}:8980/opennms/rest/nodes">
    <request method="GET">
      <parameter name="disable-ssl-verification" value="true"/>
      <parameter name="use-system-proxy" value="true"/>
    </request>
    <import-groups>xml-datacollection/opennms-nodes.xml</import-groups>
  </xml-source>

The request element can have the following optional child <parameter> elements:

Parameter Description Default

timeout

The connection and socket timeout in milliseconds.

n/a

retries

How often to repeat the request in case of an error.

0

disable-ssl-verification

Do not attempt to verify the name on the SSL certificate against the xml-source URL attribute.

false

use-system-proxy

Should the system-wide proxy settings be used? Configure the system proxy settings via system properties.

false

xslt-source-file

Full path to XSLT file to transform XML data before processing to collection definitions.

n/a

trust-store

Path to a keystore file that contains the CA or server certificates to trust when the source URL uses HTTPS. When not set, the server certificate is validated against the JVM’s default trust store.

n/a

trust-store-type

Type of the truststore file (for example, PKCS12 or JKS).

PKCS12

trust-store-password

Password of the truststore file.

n/a

key-store

Path to a keystore file that contains a client certificate and its private key. When set, the certificate is presented to servers that request one (mutual TLS).

n/a

key-store-type

Type of the keystore file (for example, PKCS12 or JKS).

PKCS12

key-store-password

Password of the keystore file.

n/a

key-password

Password of the private key, if it differs from the keystore password.

The key-store-password value.

hostname-verification

Whether to verify that the server certificate matches the host in the source URL. This applies only when trust-store or key-store is set, and takes precedence over disable-ssl-verification.

true
The keystore and truststore files must be readable by the Meridian process. If a Minion performs the collection, the files must also exist at the same path on that Minion; they are not distributed automatically. To avoid storing the passwords in plain text, store them in the Secure Credentials Vault and reference them as ${scv:alias:key}. When key-store is set, HTTP redirects are followed only when the target keeps the same scheme, host, and port as the source URL, so that the client certificate is not presented to other services. These parameters apply to the JSON collector as well, which uses the same HTTP request mechanism.

The XSLT stylesheet must be self-contained. Because the collected XML can come from an untrusted endpoint, the transformer blocks all external references to prevent XML External Entity (XXE) and server-side request forgery attacks: xsl:include, xsl:import, the document() function, and external DTDs referenced by the collected XML are not resolved. Secure processing is also enabled, which disables XSLT extension functions (for example, EXSLT or java: extensions).

HTTP headers

If the endpoint being collected requires additional headers, you can add them as part of the <request> object.

  <xml-source url="https://{nodelabel}/api/path/to/endpoint">
    <request method="GET">
      <header name="Authorization" value="Bearer static_token_here" />
      <header name="X-Api-Key" value="api-key-here" />
    </request>
    <import-groups>xml-datacollection/file.xml</import-groups>
  </xml-source>

Retrieve data from a POST request

If your XML data needs to be retrieved via a POST instead of a GET, you can specify the body content of the request.

application/form-urlencoded
  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/x-www-form-urlencoded'><![CDATA[
        <form-fields>
          <form-field name='firstName'>John</form-field>
          <form-field name='lastName'>Doe</form-field>
        </form-fields>
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>
application/xml
  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/xml'><![CDATA[
        <person>
          <firstName>John</firstName>
          <lastName>Doe</lastName>
        </person>
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>
application/json
  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/json'><![CDATA[
        {
          person: {
            firstName: 'John',
            lastName: 'Doe'
          }
        }
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

Mapping values

Sometimes data is represented as string values. These values are normally not persisted as time-series data; this means changes are not visible over time. To circumvent this, we allow mappings defined between input values and values to be persisted.

Let’s assume we have the following data input:

<records>
    <record>
        <input>aaa</input>
        <read>123</read>
    </record>
    <record>
        <input>bbb</input>
        <read>456</read>
    </record>
    <record>
        <input>ccc</input>
        <read>789</read>
    </record>
</records>

The following group configuration allows you to persist the input values as integer values over time:

<xml-group name="xml-mapping" resource-type="input" resource-xpath="/records/record" key-xpath="input">
    <xml-object name="input" type="GAUGE" xpath="input"> (1)
        <xml-mapping from="aaa" to="10" /> (2)
        <xml-mapping from="bbb" to="20" />
        <xml-mapping to="1000" /> (3)
    </xml-object>
    <xml-object name="read" type="GAUGE" xpath="read" />
</xml-group>
1 The data-type is altered in the xml-object element from STRING to GAUGE.
2 In this example we associate aaa to 10 and bbb to 20.
3 Define a default value by omitting the from attribute in a xml-mapping definition. In this example ccc will be associated with the default value of 1000.

Test XML collection via Karaf

With the configuration in place, you can test it using the collect command available in the Karaf shell:

opennms:collect -n 1 org.opennms.protocols.xml.collector.XmlCollector 127.0.0.1 collection=xml-opennms-nodes

Examples

See XmlCollector configuration examples for end-to-end recipes against common HTTP/JSON APIs.