The correct answer is the one already given. You likely either a) have a bug in your code leading to an infinite recursion which is usually quite easy to diagnose and fix, or b) have code which can lead to very deep recursions for example recursively traversing an unbalanced binary tree. In the latter situation, you need to alter your code to not allocate the information on the stack (i.e. to not recurse) but to instead allocate it in the heap.
For example, for an unbalanced tree traversal, you could store the nodes that will need to be revisited in a Stack data structure. For an in order traversal you would loop down the left branches pushing each node as you visited it until you hit a leaf, which you would process, then pop a node off the top of the stack, process it, then restart your loop with the right child (by just setting your loop variable to the right node.) This will use a constant amount of stack by moving everything that was on the stack to the heap in the Stack data structure. Heap is typically much more plentiful than stack.
As something that is usually an extremely bad idea, but is necessary in cases where memory use is extremely constrained, you can use pointer reversal. In this technique, you encode the stack into the structure you are traversing, and by reusing the links you are traversing, you can do this with no or significantly less additional memory. Using the above example, instead of pushing nodes when we loop, we just need to remember our immediate parent, and at each iteration, we set the link we traversed to the current parent and then the current parent to the node we are leaving. When we get to a leaf, we process it, then go to our parent and then we have a conundrum. We don't know whether to correct the left branch, process this node, and continue with the right branch, or to correct the right branch and go to our parent. So we need to allocate an extra bit of information as we iterate. Typically, for low-level realizations of this technique, that bit will be stored in the pointer itself leading to no additional memory and constant memory overall. This is not an option in Java, but it may be possible to squirrel away this bit in fields used for other things. In the worst-case, this is still at least 32 or 64 times reduction in the amount of memory needed. Of course, this algorithm is extremely easy to get wrong with completely confusing results and would raise utter havoc with concurrency. So it's almost never worth the maintenance nightmare except where allocating memory is untenable. The typical example being a garbage collector where algorithms like this are common.
What I really wanted to talk about, though, is when you might want to handle the StackOverflowError. Namely to provide tail call elimination on the JVM. One approach is to use trampoline style where instead of performing a tail call you return a nullary procedure object, or if you are just returning a value you return that. [Note: this requires some means of saying a function returns either A or B. In Java, probably the lightest way to do this is to return one type normally and throw the other as an exception.] Then whenever you call a method, you need to do a while loop calling the nullary procedures (which will themselves return either a nullary procedure or a value) until you get a value. An endless loop will become a while loop that is constantly forcing procedure objects that return procedure objects. The benefits of trampoline style is that it only uses a constant factor more stack than you would use with an implementation that properly eliminated all tail calls, it uses the normal Java stack for non-tail calls, the translation simple, and it only grows the code by a (tedious) constant factor. The drawback is you allocate an object on every method call (which will immediately become garbage) and consuming these objects involves a couple of indirect calls per tail call.
The ideal thing to do would be to never allocate those nullary procedures or anything else in the first place, which is exactly what tail call elimination would accomplish. Working with what Java provides though, what we could do is run the code as normal and only make these nullary procedures when we run out of stack. Now we still allocate those useless frames, but we do so on the stack rather than the heap and deallocate them in bulk, also, our calls are normal direct Java calls. The easiest way to describe this transformation is to first rewrite all multi-call-statement methods into methods that have two call statements, i.e. fgh() { f(); g(); h(); } becomes fgh() { f(); gh(); } and gh(){ g(); h(); }. For simplicity, I'll assume all methods end in a tail call, which can be arranged by just packaging the remainder of a method into a separate method, though in practice, you'd want to handle these directly. After these transformations we have three cases, either a method has zero calls in which case there is nothing to do, or it has one (tail) call, in which case we wrap it in a try-catch block in the same we will for the tail call in the two call case. Finally, it may have two calls, a non-tail call and a tail call, in which case we apply the following transformation illustrated by example (using C#'s lambda notation which could easily be replaced with an anonymous inner class with some growth):
// top-level handler
Action tlh(Action act) {
return () => {
while(true) {
try { act(); break; } catch(Bounce e) { tlh(() => e.run())(); }
}
}
}
gh() {
try { g(); } catch(Bounce e) {
throw new Bounce(tlh(() => {
e.run();
try { h(); } catch(StackOverflowError e) {
throw new Bounce(tlh(() => h());
}
});
}
try { h(); } catch(StackOverflowError e) {
throw new Bounce(tlh(() => h()));
}
}
The main benefit here is if no exception is thrown, this is the same code as we started with just with some extra exception handlers installed. Since tail calls (the h() call) don't handle the Bounce exception, that exception will fly through them unwinding those (unnecessary) frames from the stack. The non-tail calls catch the Bounce exceptions and rethrow them with the remaining code added. This will unwind the stack all the way up to the top level, eliminating the tail call frames but remembering the non-tail call frames in the nullary procedure. When we finally execute the procedure in the Bounce exception at the top-level, we will recreate all the non-tail call frames. At this point, if we immediately run out of stack again, then, since we don't reinstall the StackOverflowError handlers, it will go uncaught as desired, since we really are out of stack. If we get a little further, a new StackOverflowError will be installed as appropriate. Furthermore, if we do make progress, but then do run out of stack again, there is no benefit re-unwinding the frames we already unwound, so we install new top-level handlers so that the stack will only be unwound up to them.
The biggest problem with this approach is that you'll probably want to call normal Java methods and you may have arbitrarily little stack space when you do, so they may have enough space to start but not finish and you can't resume them in the middle. There are at least two solutions to this. The first is to ship all such work to a separate thread which will have it's own stack. This is pretty effective and pretty easy and won't introduce any concurrency (unless you want it to.) Another option is simply to purposely unwind the stack before calling any normal Java method by simply throwing a StackOverflowError immediately before them. If it still runs out of stack space when you resume, then you were screwed to begin with.
A similar thing can be done to make continuations just-in-time too. Unfortunately, this transformation isn't really bearable to do by hand in Java, and is probably borderline for languages like C# or Scala. So, transformations like this tend to be done by languages that target the JVM and not by people.
