Sunspot-Solr slowing down to a beast once my Application climbed to > 1000 objects [ Solr Logs Included ]

Question

I\'m curious if anyone noticed any scaling issues with Sunspot-Solr . Even if I remove all the searchable params, and its just counting against the raw class by itself; it still

Answer 1

1000 objects is child's play for Solr, so there's something fishy going on here with the ~200ms Solr read. However, your most immediate problem is that you're writing to Solr during what appears to be a GET request -- what's up with that? Are you saving a searchable object, which is triggering Sunspot's auto-index? If you have the need to update models during the course of a GET request (which should probably be done in a background job if possible), you'll want to disable auto-indexing in Sunspot:

searchable :auto_index => false
  # sunspot setup
end

Then you'd need to explicitly call my_model.index in your controllers when you actually do want to update them in Solr.

Finally, that big update at the end is a Solr commit, which tells Solr to write unstaged changes to disk and load up a new searcher that reflects those changes. Commits are expensive; Sunspot::Rails by default performs a commit at the end of any request that writes to Solr, but this behavior is targeted at the principle of least surprise for new users of Sunspot rather than a live app in production. You'll want to disable it in your config/sunspot.yml:

auto_commit_after_request: false

You'll then probably want to configure autoCommit in your solr/conf/solrconfig.xml -- it's commented out in the default Sunspot Solr distribution, and there is an explanation in there too. I've found that once a minute is a good place to start.

After making those changes, I'd see if your reads are still slow -- I think it's quite possible that the cause of that is that every time you search, your write/commit to Solr is causing it to have to load up a fresh searcher from disk. So, it can't allow any of its internal caches to warm, etc., and is generally under a tremendous amount of strain.

Hope that helps!