I am building a JSON-only application, it is basically a leaner Rails, with fewer middleware and fewer modules. This is an application being built in Rails 4.
I want to
did you measure the impact of adding tools like ActiveAdmin to the speed of your rest API? if not, this sounds like a case of premature optimization to me.
in the case that the API speed really has an impact by a library that is loaded but not used, then you could create multiple applications and share the models in a git submodule.
This blog post describes an interesting approach: https://content.pivotal.io/blog/migrating-from-a-single-rails-app-to-a-suite-of-rails-engines
They use a nearly empty app as a container and mount their actual apps as Rails Engines.