InShort: Occlusion Analysis for Explaining DNNs | by Eugen Lindwurm | Jan, 2021


Perhaps the simplest method to figure out why your model fails.

Eugen Lindwurm

There are abundantly many explanation methods for explaining deep neural networks (DNNs), each with its advantages and disadvantages. In most cases, we are interested in local explanation methods, i.e. explanations of the network’s output for a particular input, because DNNs tend to be too complex to be explained globally (independent of an input).

Occlusion analysis calculates the importance of every patch by observing the change of model output y when the patch is removed. The individual results can be assembled to an attribution map. Image by author

If your dimensions are independent, then occlusion analysis is perfectly faithful, as you are exactly measuring the marginal effect of each dimension.

But what does it actually mean to remove a dimension? Our model always takes inputs of the same size, after all. Removing a dimension means setting it to a value that has “0 information”. It depends on the dataset what this value is. For image data, we usually use the average RGB value. For other data types, often setting the dimension to 0 works. We will see additional considerations later.

There is another problem with occlusion analysis that is not talked about much: distribution shift (cf. Hooker et al.). If we think about it closely, the change in output that we observe in the analysis can have another reason besides information being removed: the perturbed input is no longer in the data distribution we trained the model on.

Due to its computational cost, occlusion analysis is certainly not a tool for every occasion but there are certainly uses. Especially if your data are small or you just want something that is easy to implement and reliable (just be careful with the patch size), occlusion analysis can shine. A closely related and more sophisticated approach are Shapley values. Unfortunately, they are even costlier to compute. If you are working with a differentiable model, perhaps the next-best simple approaches are gradient-based explanation methods.

Read More …


Write a comment