# Deep Learning Engine

The Deep Learning Engine is a Cluster Engine that uses a TensorFlow model to build clusters.

In order to build clusters using TensorFlow, we reduce the cluster problem to a binary classification problem that is analogous to asking: are these two alarms related? When two alarms are related, we add these to a cluster along with any other alarms that are related to any of these and so on. In order to help reduce the computational complexity in performing these pairwise computations, we limit the search of potential candidates to those that are nearby (within a distance of $\epsilon$.)

The current model definition has been developped using Ludwig:

The model input features include:

• The inventory object types (categorical)

• Relations between the inventory objects (binary)

• Difference in time (numerical)

• Distance on the graph (numerical)

 See ludwig_model.yaml for the complete model definition.

## Using the engine

To use the engine, install the `alec-engine-deeplearning` feature.

Once installed, restart the driver from the Karaf shell using:

``bundle:restart org.opennms.alec.driver.main``

Running the `list-graphs` command should now reflect that the `deeplearning` engine is being used:

``````admin@opennms> opennms-alec:list-graphs
deeplearning: 2 situations on 976 vertices and 839 edges.``````
 Once installed add the feature to the `alec.boot` in a new line with `alec-engine-deeplearning wait-for-kar=opennms-alec-plugin` to ensure that engine is re-installed when the services are restarted.

## Training

 These instructions are here to help you get started but are by no means complete. We plan to help make this process easier in future releases.

Install the following features to access the shell commands required for training:

``feature:install alec-features-deeplearning alec-features-shell``

### Vectorize the datasets

Let’s take a snapshot of the current graph:

``opennms-alec:datasource-snapshot /tmp/snap1``

Now edit `/tmp/snap1/alec.situations.xml` to reflect the desired state of situations.

Build vectors from the dataset:

``````opennms-alec:tensorflow-vectorize --alarms-in /tmp/snap1/alec.alarms.xml \
--inventory-in /tmp/snap1/alec.inventory.xml \
--situations-in /tmp/snap1/alec.situations.xml \
--csv-out /tmp/snap1/alec.vector.dataset.csv``````

### Train with Ludwig

Use Ludwig to train the model:

``ludwig train --data_csv /tmp/snap1/alec.vector.dataset.csv --model_definition_file model.yaml``
 Pull `model.yaml` from the `ludwig_model.yaml` file in the source tree

### Exporting the trained model

Adapt the following script to export the model to a format that can be used in ALEC:

``````echo '#!/usr/bin/env python
import numpy as np
import tensorflow as tf
from tensorflow.python.framework import graph_util
from tensorflow.python.framework import ops
from tensorflow.python.saved_model import builder as saved_model_builder
from ludwig import LudwigModel

model_path = "results/experiment_run_0/model"

builder = tf.saved_model.Builder("export")
with tf.Session(graph=model.model.graph) as sess:
saver = tf.train.Saver()
saver.restore(sess, model.model.weights_save_path)
builder.save()

model.close()' > export_model.py
chmod +x export_model.py
./export_model.py
mkdir -p /tmp/tf-export
cp -R ./export/* /tmp/tf-export/
cp results/experiment_run_0/model/model_hyperparameters.json /tmp/tf-export/``````

### Use the trained model in ALEC

Verify that the model can be loaded:

``opennms-alec:tensorflow-load-model /tmp/tf-export``

Configure the engine to use the model:

``````config:edit org.opennms.alec.engine.deeplearning
property-set modelPath /tmp/tf-export
config:update``````

### Run simulations using the trained model

Generate situations:

``````opennms-alec:process-alarms --alarms-in /tmp/snap1/alec.alarms.xml \
--inventory-in /tmp/snap1/alec.inventory.xml \
--situations-out /tmp/snap1/alec.situations.deeplearning.trained.xml \
--engine deeplearning``````

Compare results:

``opennms-alec:score-situations -s peer /tmp/snap1/alec.situations.xml /tmp/snap1/alec.situations.deeplearning.trained.xml``