Flutter InteractiveViewer with CustomPaint

Question

When scaling and moving with the InteractiveViewer the paint method inside CustomPaint is triggered. How to prevent that?

.