Following up on this question, it seems that a file- or disk-based Map implementation may be the right solution to the problems I mentioned there.
Map
We have a similar solution implemented using Xapian. It's fast, it's scalable, it provedes almost all search functionality you requested, it's free, multiplatform, and of course purgeable.