I have been investigating open source distributed caches and have focused on EHCahe and JBoss Pojo Cache.
They are both decent products, however the deal breaker is that my use case requires me to distribute an object graph and maintain shared references on all my nodes. EHCache cannot handle this. It uses standard java serialisation, and as such, shared references are lost.
e.g. if I have manager/employee object hierarchy, i want a single 'manager' object on all my nodes, with references to their employees.
With EHCache, I dont get this, as the java serialiser copies all dependent objects. With JBoss Pojo Cache, my object graph is preserved.
The downside is that JBoss needs to instrument your classes (the primitives that make up your object graph are stored in a tree structure behind the scenes). This can cause some issues when running in an OSGi environment.
JBoss uses JGroups as its communication stack, you can choose between multicast or TCP/IP to manage communication between the nodes.
Hope this helps.