Dynamic and Lexical variables in Common Lisp

Question

I am reading the book 'Practical Common Lisp' by Peter Seibel.

In Chapter 6, "Variables" sections "Lexical Variables and Closures" and "Dynamic, a.k.a. Special, Variables". http://www.gigamonkeys.com/book/variables.html

My problem is that the examples in both sections show how (let ...) can shadow global variables and doesn't really tell the difference between the Dynamic and Lexical vars.

I understand how closures work but I don't really get whats so special about let in this example:

(defvar *x* 10)

(defun foo ()
  (format t "Before assignment~18tX: ~d~%" *x*)
  (setf *x* (+ 1 *x*))
  (format t "After assignment~18tX: ~d~%" *x*))


(defun bar ()
  (foo)
  (let ((*x* 20)) (foo))
  (foo))


CL-USER> (foo)
Before assignment X: 10
After assignment  X: 11
NIL


CL-USER> (bar)
Before assignment X: 11
After assignment  X: 12
Before assignment X: 20
After assignment  X: 21
Before assignment X: 12
After assignment  X: 13
NIL

I feel like there is nothing special is going on here. The outer foo in bar increments the global x, and foo surrounded by let in bar increments the shadowed x. What's the big deal? I don't see how is this supposed to explain the difference between lexical and dynamic vars. Yet the book continues like this:

So how does this work? How does LET know that when it binds x it's supposed to create a dynamic binding rather than a normal lexical binding? It knows because the name has been declared special.12 The name of every variable defined with DEFVAR and DEFPARAMETER is automatically declared globally special.

What whould happen if let would bind x using "normal lexical binding"? All in all, what are the differences between dynamic and lexical binding and how is this example special regarding dynamic binding?

Answer 1

When a variable is lexically scoped, the system looks to where the function is defined to find the value for a free variable. When a variable is dynamically scoped, the system looks to where the function is called to find the value for the free variable. Variables in Common Lisp are all lexical by default; however, dynamically scoped variables can be defined at the top level using defvar or defparameter.

A simpler example

lexical scoping (with setq):

(setq x 3)

(defun foo () x)

(let ((x 4)) (foo)) ; returns 3

dynamic scoping (with defvar):

(defvar x 3)

(defun foo () x)

(let ((x 4)) (foo)) ; returns 4

How does the let know if a variable is lexical or dynamic? It doesn't. On the other hand, when foo goes to find the value of X, it will initially find the lexical value defined at the top level. It then checks to see if the variable is supposed to be dynamic. If it is, then foo looks to the calling environment, which, in this case, uses let to overshadow the value of X to be 4.

(note: this is an oversimplification, but it will help to visualize the difference between the different scoping rules)

Answer 2

What's going on?

You say: feel like there is nothing special is going on here. The outer foo in bar increments the global x, and foo surrounded by let in bar increments the shadowed x. What's the big deal?

The special that's going on here is that LET can shadow the value of *x*. With lexical variables that's not possible.

The code declares *x* to be special via the DEFVAR.

In FOO now the value of *x* is looked up dynamic. FOO will take the current dynamic binding of *x* or, if there is none, the symbol value of the symbol *x*. A new dynamic binding can, for example, be introduced with LET.

A lexical variable on the other hand has to be present in the lexical environment somewhere. LET, LAMBDA, DEFUN and others can introduce such lexical variables. See here the lexical variable x introduced in three different ways:

(let ((x 3))
  (* (sin x) (cos x)))

(lambda (x)
  (* (sin x) (cos x)))

(defun baz (x)
  (* (sin x) (cos x)))

If our code were:

(defvar x 0)

(let ((x 3))
  (* (sin x) (cos x)))

(lambda (x)
  (* (sin x) (cos x)))

(defun baz (x)
  (* (sin x) (cos x)))

Then X were special in all three above cases, because of the DEFVAR declaration, which declares X to be special - globally for all levels. Because of this, there is the convention to declare special variables as *X*. Thus only variables with stars around them are special - by convention. That's a useful convention.

In your code you have then:

(defun bar ()
  (foo)
  (let ((*x* 20))
    (foo))
  (foo))

Since *x* has be declared special via the DEFVAR above in your code, the LET construct introduces a new dynamic binding for *x*. FOO is then called. Since inside FOO the *x* uses dynamic binding, it looks up the current one and finds that *x* is dynamically bound to 20.

The value of a special variable is found in the current dynamic binding.

Local SPECIAL declarations

There are also local special declarations:

(defun foo-s ()
  (declare (special *x*))
  (+ *x* 1))

If the variable had been declared special by a DEFVAR or DEFPARAMETER, then the local special declaration can be omitted.

A lexical variable directly references the variable binding:

(defun foo-l (x)
  (+ x 1))

Let's see it in practice:

(let ((f (let ((x 10))
           (lambda ()
             (setq x (+ x 1))))))
  (print (funcall f))    ; form 1
  (let ((x 20))          ; form 2
    (print (funcall f))))

Here all variables are lexical. In form 2 the LET will not shadow the X in our function f. It can't. The function uses the lexical bound variable, introduced by the LET ((X 10). Surrounding the call with another lexically bound X in form 2 has no effect on our function.

Let's try special variables:

(let ((f (let ((x 10))
           (declare (special x))
           (lambda ()
             (setq x (+ x 1))))))
  (print (funcall f))    ; form 1
  (let ((x 20))          ; form 2
    (declare (special x))
    (print (funcall f))))

What now? Does that work?

It does not!

The first form calls the function and it tries to look up the dynamic value of X and there is none. We get an error in form 1: X is unbound, because there is no dynamic binding in effect.

Form 2 would work, since the LET with the special declaration introduces a dynamic binding for X.

Answer 3

Maybe this example will help.

;; the lexical version

(let ((x 10)) 
  (defun lex-foo ()
    (format t "Before assignment~18tX: ~d~%" x)
    (setf x (+ 1 x))
    (format t "After assignment~18tX: ~d~%" x)))

(defun lex-bar ()
  (lex-foo)
  (let ((x 20)) ;; does not do anything
    (lex-foo))
  (lex-foo))

;; CL-USER> (lex-bar)
;; Before assignment X: 10
;; After assignment  X: 11
;; Before assignment X: 11
;; After assignment  X: 12
;; Before assignment X: 12
;; After assignment  X: 13

;; the dynamic version

(defvar *x* 10)
(defun dyn-foo ()
  (format t "Before assignment~18tX: ~d~%" *x*)
  (setf *x* (+ 1 *x*))
  (format t "After assignment~18tX: ~d~%" *x*))

(defun dyn-bar()
  (dyn-foo)
  (let ((*x* 20))
    (dyn-foo))
  (dyn-foo))

;; CL-USER> (dyn-bar)
;; Before assignment X: 10
;; After assignment  X: 11
;; Before assignment X: 20
;; After assignment  X: 21
;; Before assignment X: 11
;; After assignment  X: 12

;; the special version

(defun special-foo ()
  (declare (special *y*))
  (format t "Before assignment~18tX: ~d~%" *y*)
  (setf *y* (+ 1 *y*))
  (format t "After assignment~18tX: ~d~%" *y*))

(defun special-bar ()
  (let ((*y* 10))
    (declare (special *y*))
    (special-foo)
    (let ((*y* 20))
      (declare (special *y*))
      (special-foo))
    (special-foo)))

;; CL-USER> (special-bar)
;; Before assignment X: 10
;; After assignment  X: 11
;; Before assignment X: 20
;; After assignment  X: 21
;; Before assignment X: 11
;; After assignment  X: 12

Answer 4

You can tell your Lisp to bind local variables dynamically, too:

(let ((dyn 5))
  (declare (special dyn))
  ... ;; DYN has dynamic scope for the duration of the body
  )

Answer 5

Rewrite example from PCL.

;;; Common Lisp is lexically scoped by default.

λ (setq x 10)
=> 10

λ (defun foo ()
    (setf x (1+ x)))
=> FOO

λ (foo)
=> 11

λ (let ((x 20))
    (foo))
=> 12

λ (proclaim '(special x))
=> NIL

λ (let ((x 20))
    (foo))
=> 21

Yet another great explanation from On Lisp, chapter 2.5 Scope:

Common Lisp is a lexically scoped Lisp. Scheme is the oldest dialect with lexical scope; before Scheme, dynamic scope was considered one of the defining features of Lisp.

The difference between lexical and dynamic scope comes down to how an implementation deals with free variables. A symbol is bound in an expression if it has been established as a variable, either by appearing as a parameter, or by variable-binding operators like let and do. Symbols which are not bound are said to be free. In this example, scope comes into play:

(let ((y 7)) 
  (defun scope-test (x)
  (list x y)))

Within the defun expression, x is bound and y is free. Free variables are interesting because it’s not obvious what their values should be. There’s no uncertainty about the value of a bound variable—when scope-test is called, the value of x should be whatever is passed as the argument. But what should be the value of y? This is the question answered by the dialect’s scope rules.

In a dynamically scoped Lisp, to find the value of a free variable when exe- cuting scope-test, we look back through the chain of functions that called it. When we find an environment where y was bound, that binding of y will be the one used in scope-test. If we find none, we take the global value of y. Thus, in a dynamically scoped Lisp, y would have the value it had in the calling expression:

> (let ((y 5)) (scope-test 3))
    (3 5)

With dynamic scope, it means nothing that y was bound to 7 when scope-test was defined. All that matters is that y had a value of 5 when scope-test was called.

In a lexically scoped Lisp, instead of looking back through the chain of calling functions, we look back through the containing environments at the time the function was defined. In a lexically scoped Lisp, our example would catch the binding of y where scope-test was defined. So this is what would happen in Common Lisp:

> (let ((y 5)) (scope-test 3))
    (3 7)

Here the binding of y to 5 at the time of the call has no effect on the returned value.

Though you can still get dynamic scope by declaring a variable to be special, lexical scope is the default in Common Lisp. On the whole, the Lisp community seems to view the passing of dynamic scope with little regret. For one thing, it used to lead to horribly elusive bugs. But lexical scope is more than a way of avoiding bugs. As the next section will show, it also makes possible some new programming techniques.