Update 3
I updated the code (http://codepen.io/anon/pen/VYRJLp) and now the edges look really like torn paper effect. It uses SVG by the way.
The
You could probably achieve this effect with a png image.
The top part should be black (or whatever the color of your header), the bottom is partly transparent to create the "torn effect", and your real image will be on a lower layer (z-index) to be partly hidden.
Reply to Update2 this is what I meant
Instead of creating a complete shell (a "square" hole), you can split the four edges in four images: top, right, bottom and left.
You can make it responsive by simply repeating the image on the whole length (the "start" and "end" of the four edges must be the same) *