# How can I interpret the way the neural network is producing an output for a given input?

I'm using a small neural network (2 hidden layers, 60 neurons apiece) for a rather complex binary classification problem.

The network works well, but I'd like to know how it is using the inputs to perform the classification. Ultimately, I would like to interpret the trained network in order to learn more about the processes responsible for generating the data.

Ideally, I would end up with an equation that would allow me to perform the classification without the network and that would have parameters that I could interpret in the context of the system the network is being used on.

My first thought is to procedurally mask out a growing subset of the ~4000 parameters until there's an appropriate trade-off between performance and simplicity and then maybe use a symbolic logic library to try and simplify further.

I don't think that's the best plan, so I wonder if there's an existing workflow to interpret a neural network.

• It's not clear to me what you're really asking here. Are you asking how to represent this specific neural network as an equation, so something like $f(x) = 2x^2 + 3$?
– nbro
Aug 19 at 14:17
• @nbro I've edited the question to hopefully make it less vague. The data were generated from a simulation of a biological process that I would like to learn more about. I would like to use not the architecture, but the trained model to ideally generate an equation that relates 15 inputs to one binary output. Aug 19 at 23:00
• Thanks! Your last edit clarifies what you are asking. I tried to make the title the question you're asking. Make sure that's the case. If not, feel free to edit the post again.
– nbro
Aug 19 at 23:04
• Having said that, the answer to your question may be in one of the answers to this question. I will not mark yours as a duplicate, because it seems to me that your problem/question is more specific.
– nbro
Aug 19 at 23:05
• If you're using something like PyTorch or TensorFlow, you can also create synthetic samples using gradient feedback loops which creates saliency maps showing what features really might help triggering your classification. See this: github.com/utkuozbulak/pytorch-cnn-visualizations Aug 20 at 15:39

"Ideally, I would end up with an equation that would allow me to perform the classification without the network".

If you could find such analytic equation without machine learning then why training a multi layer perceptron in the first place? Or to phrase it differently, the mlp you trained is that equation. And I'm not trying to be ironic, if you need an analytic explanation then don't use a multi layer perceptron, but move to decision trees algorithms for example, than you could literally plot the model itself (still hard to interpret depending on the amount of features you're using).

If instead you still want to stick with mlp, then something that you could do to better understand your model is plotting the decision boundaries learned by it. Sklearn has a nice tutorial on how to do it, I copied it and changes the svm with an mlp just to show that the approach works regardless of the model

import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.neural_network import MLPClassifier

# import some data to play with
X = iris.data[:, :2]  # we only take the first two features. We could
# avoid this ugly slicing by using a two-dim dataset
y = iris.target

h = 0.02  # step size in the mesh

# we create an instance of SVM and fit out data. We do not scale our
# data since we want to plot the support vectors
C = 1.0  # SVM regularization parameter
mlp = MLPClassifier().fit(X, y)

# create a mesh to plot in
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

# title for the plots
title = "MLP boundries"

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, x_max]x[y_min, y_max].
# plt.subplot(1, 1)

Z = mlp.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.8)

# Plot also the training points
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.coolwarm)
plt.xlabel("Sepal length")
plt.ylabel("Sepal width")
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xticks(())
plt.yticks(())
plt.title(title)

plt.show()


The code output this plot:

Your can see that the boundaries are a crude approximation of the mlp behavior, they are just estimated from a brute force prediction applied to all points of the 2D graph generated by 2 features of the input data. SO the boundaries will change depending also on the features you decide to plot. But it gives you an idea about the relationships learned by the mlp.

If you want more, I stress again that you should train a different model, like decision tree, random forest or XGboost, with these model you can compute scores about features importance and literally plot the decision thresholds learned by the models.