Editing programs “while they are running”? How?

Question

This question is a corollary to: Editing programs “while they are running”? Why?

I\'m only recently being exposed to the world of Clojure and am fascinated by a few exam

Answer 1

There are a lot of good answers here, and I'm not sure I can improve on any of them, but I wanted to add some comments around Clojure and Java.

First off, Clojure is written in Java, so you can definitely build a live-coding environment in Java. Just think of Clojure as a specific flavor of live-coding environment.

Basically, live coding in Clojure works via the read function in main.clj and the eval function in core.clj (src/clj/clojure/main.clj and src/clj/clojure/core.clj in the github repository). You read in the forms and pass them to eval, which calls the clojure.lang.Compiler (src/jvm/clojure/lang/Compiler.java in the repo).

Compiler.java converts Clojure forms into JVM bytecode using the ASM library (ASM website here, documentation here). I'm not sure what version of the ASM library is used by Clojure. This bytecode (an array of bytes => byte[] bytecode is the member of the Compiler class that will ultimately hold the bytes generated by the clojure.asm.ClassWriter class via ClassWriter#toByteArray) must then be converted to a class and linked into the running process.

Once you have a representation of a class as a byte array, it's a matter of getting ahold of a java.lang.ClassLoader, calling defineClass to turn those bytes into a Class, and then passing the resulting Class to the resolve method of the ClassLoader to link it to the Java runtime. This is basically what happens when you define a new function, and you can see the internals of the compiler in Compiler$FnExpr which is the inner class that generates the bytecode for function expressions.

There's more going on than that with respect to Clojure, such as the way in which it handles namespace and symbol interning. I'm not completely sure how it gets around the fact that the standard ClassLoader will not replace a linked Class with a new version of that Class, but I suspect it has to do with how classes are named and how symbols are interned. Clojure also defines its own ClassLoader, a certain clojure.lang.DynamicClassLoader, which inherits from java.net.URLClassLoader, so that might have something to do with it; I'm not sure.

In the end, all the pieces are there to do live-coding in Java between ClassLoaders and bytecode generators. You just have to provide a way to input forms into a running instance, eval the forms, and link them up.

Hope this sheds a little more light on the subject.

Answer 2

Some language implementations have that for a long time, especially many Lisp variants and Smalltalk.

Lisp has identifiers as a data structure, called symbols. These symbols can be reassigned and they are looked up at runtime. This principle is called late binding. Symbols name functions and variables.

Additionally Lisp implementations either have at runtime an interpreter or even a compiler. The interface are the functions EVALand COMPILE. Plus there is a function LOAD, which allows loading of source code and compiled code.

Next a language like Common Lisp has an object system which allows changes to the class hierarchy, classes themselves, can add/update/remove methods and propagates these changes to already existing objects. So the object-oriented software and code can be updated itself. With the Meta-object Protocol one can even re-program the object system at runtime.

It's also important that Lisp implementations then can garbage collect removed code. That way the running Lisp will not grow in runtime size just because code is replaced.

Lisp often also has an error system which can recover from errors and allows replacing defective code from within the debugger.

Answer 3

All that is required is:

• the language must have the ability to load new code (eval)
• an abstraction to redirect function/method calls (vars or mutable-namespaces)

Answer 4

The concepts originated in the Lisp world, but just about any language can do it (certainly, if you have a repl, you can do this sort of stuff). It's simply better known in the Lisp world. I know there are slime-esque packages for haskell and ruby, and I would be very surprised if such a thing didn't exist for Python as well.

Answer 5

JRebel is one solution for Java. Here's a brief passage from their FAQ:

JRebel integrates with the JVM and application servers mainly on the class loader level. It does not create any new class loaders, instead, it extends the existing ones with the ability to manage reloaded classes.

Answer 6

It is a pattern that can be applied to any language, provided that the language was written with an environment that permits reassigning names associated with blocks of code.

In the computer, code and data exists in memory. In programming languages, we use names to refer to those "chunks" of memory.

int a = 0;

would "name" some number of bytes of memory "a". It would also "assign" that memory the byte value corresponding to 0. Depending on the type system,

int add(int first, int second) {
  return first + second;
}

would "name" some number of bytes of memory "add". It would also "assign" that memory to contain the machine instructions to look into the call stack for two "int" numbers, add them together, and put the result in the appropriate place on the call stack.

In a type system that separates (and maintains) names to blocks of code, the end result is that you can easily pass blocks of code around by reference much in the same way you can variable memory around by reference. The key is to make sure the type system "matches" only compatible types, otherwise passing around the blocks of code might induce errors (like returning a long when originally defined to return an int).

In Java, all types resolve to a "signature" which is a string representation of the method name and "type". Looking at the add example provided, the signature is

// This has a signature of "add(I,I)I"
int add(int first, int second) {
  return first + second;
}

If Java supported (as Clojure does) method name assignment, it would have to expand on its declared type system rules, and allow method name assignment. A fake example of method assignment would logically look like

subtract = add;

but this would require the need to declare subtract, with a strongly typed (to match Java) "type".

public subtract(I,I)I;

And without some care, such declarations can easily tread upon already-defined parts of the language.

But to get back to your answer, in languages that support such, the names basically are pointers to blocks of code, and can be reassigned provided you don't break the expectations of input and return parameters.