Is there any way to ensure key material gets securely erased from the memory after the program exits? Being able to erase it manually and keep the program running would be even
GHC can return memory to the OS when it is no longer needed, so merely blanking the memory on exit won't achieve your goal. Garbage collection is a complicated business, but there is in general no way to ensure that old copies of your secure data are not returned to the OS memory pool.
However the OS will blank your memory before allocating it to another process. If you don't trust the OS to keep your memory secure then you have a much bigger problem.
I'm not sure what you mean by "unreliable"; Haskell GC is reliable, but the program has comparatively little visibility of what is happening.
However if you are concerned merely with a cryptographic key rather than a big, complicated data structure then life gets a bit better. You can use a Foreign Pointer to point to a memory location for your key, and then make blanking that bit of memory into a part of your finaliser. You can even write a bit of code that allocates a block of memory, mlocks it, and then hands off foreign pointers to key-sized chunks of that memory on request, with finalisers that wipe the key. That would probably do what you want.
The point of a ForeignPtr is that it is guaranteed not to be moved or re-interpreted by the GC.