Adaptive Activation Thresholding: Dynamic Routing Type Behavior for Interpretability in Convolutional Neural Networks

Yiyou Sun, Sathya Ravi, Vikas Singh

Why we propose interpretable CNN?

Most techniques that focus on interpretability approach the problem in a post hoc manner.
While such approaches are very useful to better interpret or debug the model’s behavior, the explanation emerging from various techniques may be incomplete and inaccurate[1]. Instead, what is suggested is to build a model that is, by design, interpretable.

Fig 1(a): The large residual(unexplainable) part exists in IBD method[2], which is a framework for providing explanations to networks for visual recognition.

Fig 1(b): The connection logic visualized by [3] can be confusing, because the visualization of each unit is vague.

What is Dynamic Connection? How does Dynamic Connection help interpretablity?

The dynamic connection is a type of connection between layers that associates a higher-level unit utilizing only a small support of lower-level units. In other words, the route can be considered “dynamic” – at the level of classes or even individual examples. For example, in EM routing[4], the upper-level units’ activation only depends on (or driven by) a few lower-level units which are inside the range or support of the estimated covariance matrix.
This has implications for intepretability due to following reasons[5]:
1. Sparsity. It reduces the parameters to be analyzed.
2. Simulatability. Human is able to simulate its decision-making process.
3. Modularity. The meaningful portion(units) of its prediction-making process can be interpreted independently.

Fig 2: Show that the prediction logic by dynamic connection is closer to the actual semantic representation.

How do we propose interpretable CNN?

We simplify the EM routing[4] algorithm to permit the dynamic behaviour in standard CNN architectures.

Fig 3: A simple illustration of the mechanism of dynamic connection layer. Here we present a ResNet-liked structure whose final fc layer is replaced by a the Dynamic Connection Layer.

Can such behavior coexist with accuracy? Can dynamic connection encourage sparsity?

The answer is Yes.
In Fig.4(a), we compare the train and test accuracy curve over 200 epochs of dynamic CNN (Res18-d1a, Res18-d5a) and normal CNN(Res18, Res18-L1) trained on Cifar10. Note that the accuracy of the models obtained using our algorithm and the baseline are nearly the same.
In Fig.4(b), for all models, we visualize the final fully connected layer’s weight (512 × 10) to compare the sparsity. The final layer obtained using our Dynamic-K activation algorithm is much sparser than the baseline model. (Res18-d1a > Res18-d5a >> Res18-L1 > Res18)

Fig 4(a): Training and Testing Accuracy Curve of Different ResNet18s on Cifar10
Fig 4(b): Weight Visualization of Final Layer for different ResNets.

Is model more Interpretable?

The answer is Yes.
In this example, every unit in the final convolutional layer is analyzed by NetDissect algorithm and labeled by the closest concept. Then by simply looking at the weights, we can characterize the decision boundaries of each class.


This paper proposes a simple dynamic connect/routing algorithm based on choosing several top activated lower-level units for a higher-level unit. The idea is extremely simple and inspired by the dynamic/EM routing algorithm[4]. But our simplified version can be plugged into any current traditional CNN structures and involves minimal changes to the optimization code. The idea is simple to understand, works well, yet we can rigorously analyze its convergence properties. Our experiments show that this simple scheme can help speed up convergence, encourage sparsity of the network and increase interpretability, without sacrificing the overall accuracy.