I am looking for a reasonably well tested library+server to store a persistent distributed hash table.
I am hesistant to use SQL-based solutions as the data is highly d
DryadLINQ or Hadoop.Net may help.
Hadoop.Net is dotnet version of Hadoop. More about Hadoop can be found here
Consider MS Velocity.
Summary: “Velocity” is a distributed in-memory application cache platform for developing scalable, available, and high-performance applications. “Velocity” fuses memory across multiple computers to give a single unified cache view to applications. Applications can store any serializable CLR object without worrying about where the object gets stored. Scalability can be achieved by simply adding more computers on demand. “Velocity” also allows for copies of data to be stored across the cluster, thus protecting data against failures. “Velocity” can be configured to run as a service accessed over the network or can be run embedded with the distributed application.
Take a look at Ayende's Rhino DHT. Might be more inline with what you are looking for. The source can be acquired here.
I actually think you should consider SQL Server 2008. Store the data in a table with a varbinary(max) column, along with a column that contains the hash of that column. Index the hash, as you suggested.
You'll then be able to use the various distribution features of the product.
You can try StorageEdge it has NCache technology at it's back-end so by using it you'll have the support of distributed cache which will boost SharePoint performance, reliability, scalability and optimize its storage at the same time.
Here is link to StorageEdge's homepage http://www.alachisoft.com/storageedge/ I hope it help :)
Rest for .NET you can always try NCache a big name in Distributed Caching you can find its details on http://www.alachisoft.com/ncache/