I am building a JSON-only application, it is basically a leaner Rails, with fewer middleware and fewer modules. This is an application being built in Rails 4.
I want to
did you measure the impact of adding tools like ActiveAdmin to the speed of your rest API? if not, this sounds like a case of premature optimization to me.
in the case that the API speed really has an impact by a library that is loaded but not used, then you could create multiple applications and share the models in a git submodule.