Wagtail not redirecting. 500 error instead of 404 when DEBUG = False

Question

The guy here already asked about this and solved it by deleting 500.html and 404.html.

Same for me, when DEBUG is True