In Haskell, you can build infinite lists due to laziness:
Prelude> let g = 4 : g
Prelude> g !! 0
4
Prelude> take 10 g
[4,4,4,4,4,4,4,4,4,4]
You are correct as to why it hangs--you've created a circular dependency that it can't resolve. Computing the current element requires a later element that can't be computed until the current one is computed, blah blah, around in circles we go.
As for why it doesn't produce an exception, try compiling it instead of running it in GHCi:
$ ghc --make Loop.hs
$ ./Loop.exe
Loop.exe: <>
I'm assuming this is a NonTermination exception. Pff, Halting Problem? Ha.
Not everything works the way you'd like or expect it to when done in GHCi. If something seems odd, try compiling a small example and see if it makes more sense that way. GHCi using different rules for type defaulting trips me up sometimes, for instance.