I would like to make requests to the WordPress API much faster. My API is implemented in a plugin (using register_rest_route to register my routes). However, since this is a plu
You should give this a try this. It is a plug-in that allows you to enable/disable certain plug-ins for post-types, pages and other circumstances.
For the theme part, if you wrote it, it would be easy to add something in the function.php to prevent it from attaching any hooks or filters in the case of an API request.
As a sidenote, couldn't you query de DB directly?