How to avoid all widget rebuild with setState in flutter

Question

I am trying to add a border to a container starts scrolling down and remove it if scrolled back to the top.

I am listening to the scroll controller and setting the st