Stack overflow from recursive function call in Lisp

Question

I am learning Lisp from the book \"The Land of Lisp\" by Conrad Barski. Now I have hit my first stumbling block, where the author says:

Calling yourse

Answer 1

You're looking for tail recursion. At the moment your function is defined like

(defun my-length (list)
  (if list
    (1+ (my-length (cdr list)))
    0))

Notice that after my-length has called itself, it needs to add one to the result before passing that value to its calling function. This need to modify the value before returning it means that you need to allocate a new stack frame for every call, the the space used is proportional to the length of the list. This is what causes a stack overflow on long lists.

You can re-write it to use a helper function

(defun helper (acc list)
  (if list
    (helper (1+ acc) (cdr list))
    acc))

(defun my-length (list)
    (helper 0 list))

The helper function takes two parameters, an accumulation parameter acc, which stores the length of the list so far, and a list list which is the list we're computing the length of.

The important point is that helper is written tail recursively, which means that calling itself is the last thing it does. This means you don't need to allocate a new stack frame every time the function calls itself - since the final result will just be passed all the way back up the chain of stack frames anyway, you can get away with overwriting the old stack frame with a new one, so your recursive function can operate in constant space.

This form of program transformation - from a recursive (but non-tail-recursive) definition to an equivalent definition using a tail-recursive helper function is an important idiom in functional programming - one that it's worth spending a bit of time understanding.