XmlCollector

The XmlCollector collects and extracts metrics from XML and JSON documents.

Collector facts

Class Name

org.opennms.protocols.xml.collector.XmlCollector

Package

core

Supported on Minion

Yes (see limitations)

Configuration Files

${OPENNMS_HOME}/etc/xml-datacollection-config.xml
${OPENNMS_HOME}/etc/xml-datacollection.d/*.xml

Configuration and use

Table 1. Collector-specific parameters for the XmlCollector
Parameter	Description	Default
Required
collection	The name of the XML Collection to use.	n/a
Optional
handler-class	Class that performs the collection.	org.opennms.protocols.xml.collector.DefaultXmlCollectionHandler

The available handlers include:

org.opennms.protocols.xml.collector.DefaultXmlCollectionHandler
org.opennms.protocols.xml.collector.Sftp3gppXmlCollectionHandler
org.opennms.protocols.xml.vtdxml.DefaultVTDXmlCollectionHandler
org.opennms.protocols.xml.vtdxml.Sftp3gppVTDXmlCollectionHandler
org.opennms.protocols.json.collector.DefaultJsonCollectionHandler
org.opennms.protocols.http.collector.HttpCollectionHandler

Caveats

The org.opennms.protocols.json.collector.DefaultJsonCollectionHandler requires the fetched document to be a single element of type object to make xpath query work. If the root element is an array, it will be wrapped in an object accessible as /elements.

JSON object keys are not restricted the way XML element names are: a key may start with a digit or contain characters that are not allowed in an XML name, for example 2xx or o’brien. A plain XPath location step cannot express such a name, so the JSON collection handler rewrites those steps transparently to an equivalent *[name()='…'] step before evaluating them. You can therefore write the key exactly as it appears in the JSON document:

<xml-group name="http-stats" resource-type="node" resource-xpath="/stats">
  <xml-object name="ok" type="GAUGE" xpath="2xx/count"/>
  <xml-object name="failed" type="GAUGE" xpath="5xx/count"/>
</xml-group>

This works for resource-xpath, key-xpath, timestamp-xpath and the xpath of an xml-object, and predicates keep working on a rewritten step, for example 2xx[1]/count. A key that holds an array still produces one resource per array element.

The rewrite applies only to the JSON collection handler and only to steps that are not already a legal XML name. It leaves predicates, functions, axis specifiers (self::, child::), ., .., *, node() and text() untouched, which has two consequences worth knowing:

A step written with an explicit axis or namespace prefix is not rewritten — write 2xx rather than child::2xx.
A value that is a full XPath expression rather than a location path, such as a union like /a | /b, is left as-is.

XML collection configuration

Understanding resource types helps when editing collector-specific configuration files.

XML collections are defined in ${OPENNMS_HOME}/etc/xml-datacollection-config.xml and groups are defined in ${OPENNMS_HOME}/etc/xml-datacollection/.

Sample collection definition named xml-opennms-nodes

<xml-collection name="xml-opennms-nodes">
  <rrd step="300">
    <rra>RRA:AVERAGE:0.5:1:2016</rra>
    <rra>RRA:AVERAGE:0.5:12:1488</rra>
    <rra>RRA:AVERAGE:0.5:288:366</rra>
    <rra>RRA:MAX:0.5:288:366</rra>
    <rra>RRA:MIN:0.5:288:366</rra>
  </rrd>
  <xml-source url="http://admin:admin@{ipaddr}:8980/opennms/rest/nodes">
    <import-groups>xml-datacollection/opennms-nodes.xml</import-groups>
  </xml-source>
</xml-collection>

The url attribute of the <xml-source> supports node and IpInterface field placeholders.

Metadata expressions can also be used in attributes of the xml-datacollection-config.xml configuration file. This lets the user to also reference credentials stored in the secure credentials.

The referenced opennms-nodes.xml file

<xml-groups>
    <xml-group name="nodes" resource-type="node" resource-xpath="/nodes">
        <xml-object name="totalCount" type="GAUGE" xpath="@totalCount"/>
    </xml-group>
</xml-groups>

Customize the source request

If you need to customize the request to pull raw data, you can add a <request> element within the <xml-source> block.

  <xml-source url="http://admin:admin@{ipaddr}:8980/opennms/rest/nodes">
    <request method="GET">
      <parameter name="disable-ssl-verification" value="true"/>
      <parameter name="use-system-proxy" value="true"/>
    </request>
    <import-groups>xml-datacollection/opennms-nodes.xml</import-groups>
  </xml-source>

The request element can have the following optional child <parameter> elements:

Parameter Description Default

Parameter	Description	Default
timeout	The connection and socket timeout in milliseconds.	n/a
retries	How often to repeat the request in case of an error.	0
disable-ssl-verification	Do not attempt to verify the name on the SSL certificate against the `xml-source` URL attribute.	false
use-system-proxy	Should the system-wide proxy settings be used? Configure the system proxy settings via system properties.	false
xslt-source-file	Full path to XSLT file to transform XML data before processing to collection definitions.	n/a

timeout

The connection and socket timeout in milliseconds.

n/a

retries

How often to repeat the request in case of an error.

disable-ssl-verification

Do not attempt to verify the name on the SSL certificate against the xml-source URL attribute.

false

use-system-proxy

Should the system-wide proxy settings be used? Configure the system proxy settings via system properties.

false

xslt-source-file

Full path to XSLT file to transform XML data before processing to collection definitions.

n/a

The XSLT stylesheet must be self-contained. Because the collected XML can come from an untrusted endpoint, the transformer blocks all external references to prevent XML External Entity (XXE) and server-side request forgery attacks: xsl:include, xsl:import, the document() function, and external DTDs referenced by the collected XML are not resolved. Secure processing is also enabled, which disables XSLT extension functions (for example, EXSLT or java: extensions).

HTTP headers

If the endpoint being collected requires additional headers, you can add them as part of the <request> object.

  <xml-source url="https://{nodelabel}/api/path/to/endpoint">
    <request method="GET">
      <header name="Authorization" value="Bearer static_token_here" />
      <header name="X-Api-Key" value="api-key-here" />
    </request>
    <import-groups>xml-datacollection/file.xml</import-groups>
  </xml-source>

Retrieve data from a POST request

If your XML data needs to be retrieved via a POST instead of a GET, you can specify the body content of the request.

application/form-urlencoded

  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/x-www-form-urlencoded'><![CDATA[
        <form-fields>
          <form-field name='firstName'>John</form-field>
          <form-field name='lastName'>Doe</form-field>
        </form-fields>
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

application/xml

  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/xml'><![CDATA[
        <person>
          <firstName>John</firstName>
          <lastName>Doe</lastName>
        </person>
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

application/json

  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/json'><![CDATA[
        {
          person: {
            firstName: 'John',
            lastName: 'Doe'
          }
        }
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

Collect files over SFTP

The XmlCollector can read XML files from an SFTP server as well as from HTTP endpoints. Two URL schemes are available:

sftp://: This scheme retrieves one file at a fixed path on each collection and works with the default handler.
sftp.3gpp://: This scheme retrieves 3GPP performance measurement files named according to 3GPP TS 32.432. It is intended for use with the Sftp3gppXmlCollectionHandler or Sftp3gppVTDXmlCollectionHandler handler.

SFTP URLs

An SFTP source URL has the form sftp://<username>:<password>@<host>[:<port>]/<path>.

Sample collection that reads one file over SFTP

<xml-collection name="sftp-stats">
  <rrd step="300">
    <rra>RRA:AVERAGE:0.5:1:2016</rra>
  </rrd>
  <xml-source url="sftp://${scv:sftp-stats:username}:${scv:sftp-stats:password}@{ipaddr}/data/stats.xml"> (1)
    <import-groups>xml-datacollection/sftp-stats.xml</import-groups>
  </xml-source>
</xml-collection>

1	The credentials in this example are stored in the Secure Credentials Vault under the alias `sftp-stats`. They can also be written directly in the URL.

The following rules apply to SFTP URLs:

The port defaults to 22.
The path is the absolute path of the file as the SFTP server presents it. On a server that restricts users to a chroot directory, the path starts at that directory.
A username is required. The password can be left out when key-based authentication is configured.
The password can’t contain the characters @, /, ?, #, or :, and percent-encoded characters are not decoded.
The <request> parameters described above apply only to HTTP and HTTPS sources. SFTP connections use a fixed timeout of 30 seconds.

The collector does not verify the host key of the SFTP server and accepts whatever key the server presents.

Key-based authentication

To authenticate with a private key, set the system property sftp.private-key.location to the path of the key file, for example in ${OPENNMS_HOME}/etc/opennms.properties.d/sftp.properties:

sftp.private-key.location=/opt/opennms/etc/sftp/id_ed25519

The source URL then needs only a username, for example sftp://pmuser@{ipaddr}/data/stats.xml.

The property applies to every SFTP source. The key file must be readable by the Meridian process and can’t be protected with a passphrase. Changes to the property take effect after a restart. If a Minion performs the collection, the property and the key file must be configured on that Minion.

3GPP measurement files

3GPP performance measurement files are named A<date>.<start time><UTC offset>-<end time><UTC offset>_<NE ID>, for example A20260923.0905-0400-0910-0400_NE01 for the interval from 09:05 to 09:10 at UTC-4. An sftp.3gpp:// URL points to the directory that holds these files, and its query parameters describe how the file names are built:

sftp.3gpp://<username>:<password>@<host>[:<port>]/<directory>?<parameters>

Table 2. Query parameters for `sftp.3gpp://` URLs
Parameter	Description	Default
step	Length of the measurement interval in seconds. The `{step}` placeholder resolves to the `step` of the collection’s `<rrd>` element.	Required
neId	Network element ID that appears at the end of each file name.	Required
timezone	Time zone used to build the file names, as a Java time zone ID such as `America/Chicago`.	Time zone of the Meridian JVM
deleteFile	Whether to delete each file from the server after it is processed.	false

The Sftp3gppXmlCollectionHandler handler retrieves files by these rules:

On the first collection for a node, service, and directory, the handler requests the file for the most recent completed interval. The file must exist at that time, and the request uses a name without a file extension.
After it processes a file, the handler stores the file name. Later collections list the directory and process, in name order, every file whose interval ends after the stored one.
Files whose names start with . are skipped.
When deleteFile is true, only the files that the handler processed are deleted.
Each resource receives the string attributes label and instance, which hold the resource instance.

Sample collection for 3GPP measurement files

<xml-collection name="3gpp-pm">
  <rrd step="900">
    <rra>RRA:AVERAGE:0.5:1:8928</rra>
  </rrd>
  <xml-source url="sftp.3gpp://${scv:ne-pm:username}:${scv:ne-pm:password}@{ipaddr}/export/pm?step={step}&amp;neId={foreignId}&amp;timezone=America/Chicago&amp;deleteFile=true"> (1)
    <xml-group name="platform-system-resource" resource-type="platformSystemResource"
               key-xpath="@measObjLdn"
               resource-xpath="/measCollecFile/measData/measInfo[@measInfoId='platform-system|resource']/measValue"
               timestamp-xpath="/measCollecFile/fileFooter/measCollec/@endTime" (2)
               timestamp-format="yyyy-MM-dd'T'HH:mm:ssZ">
      <xml-object name="cpuUtilization" type="GAUGE" xpath="r[@p=1]"/>
      <xml-object name="memoryUtilization" type="GAUGE" xpath="r[@p=2]"/>
    </xml-group>
  </xml-source>
</xml-collection>

1	The `&` characters between query parameters are written as `&` inside the XML attribute.
2	The timestamp XPath stores each value at the end time of its measurement interval.

The resource type referenced by the group needs a definition, for example in ${OPENNMS_HOME}/etc/resource-types.d/3gpp.xml:

<resource-types>
  <resourceType name="platformSystemResource" label="3GPP Processor" resourceLabel="${label}">
    <persistenceSelectorStrategy class="org.opennms.netmgt.collection.support.PersistAllSelectorStrategy"/>
    <storageStrategy class="org.opennms.protocols.xml.collector.XmlStorageStrategy"/>
  </resourceType>
</resource-types>

The collectd service selects the handler with the handler-class parameter:

<service name="3GPP-PM" interval="900000" user-defined="false" status="on">
  <parameter key="collection" value="3gpp-pm"/>
  <parameter key="handler-class" value="org.opennms.protocols.xml.collector.Sftp3gppXmlCollectionHandler"/>
</service>

Mapping values

Sometimes data is represented as string values. These values are normally not persisted as time-series data; this means changes are not visible over time. To circumvent this, we allow mappings defined between input values and values to be persisted.

Let’s assume we have the following data input:

<records>
    <record>
        <input>aaa</input>
        <read>123</read>
    </record>
    <record>
        <input>bbb</input>
        <read>456</read>
    </record>
    <record>
        <input>ccc</input>
        <read>789</read>
    </record>
</records>

The following group configuration allows you to persist the input values as integer values over time:

<xml-group name="xml-mapping" resource-type="input" resource-xpath="/records/record" key-xpath="input">
    <xml-object name="input" type="GAUGE" xpath="input"> (1)
        <xml-mapping from="aaa" to="10" /> (2)
        <xml-mapping from="bbb" to="20" />
        <xml-mapping to="1000" /> (3)
    </xml-object>
    <xml-object name="read" type="GAUGE" xpath="read" />
</xml-group>

1	The data-type is altered in the `xml-object` element from `STRING` to `GAUGE`.
2	In this example we associate `aaa` to `10` and `bbb` to `20`.
3	Define a default value by omitting the `from` attribute in a `xml-mapping` definition. In this example `ccc` will be associated with the default value of `1000`.

Test XML collection via Karaf

With the configuration in place, you can test it using the collect command available in the Karaf shell:

opennms:collect -n 1 org.opennms.protocols.xml.collector.XmlCollector 127.0.0.1 collection=xml-opennms-nodes

To test a collection that uses a different handler, add the handler-class attribute:

opennms:collect -n 1 org.opennms.protocols.xml.collector.XmlCollector 127.0.0.1 collection=3gpp-pm handler-class=org.opennms.protocols.xml.collector.Sftp3gppXmlCollectionHandler