Is there a way to decompose a CNN model to provide prediction insights to humans?

Question

I have a CNN that after a lot of work is now performing multiclass (8) classification at 99% accuracy.

While the classification itself has a lot of value, going to a