What are the best practices to make shiny application run faster?

Question

Data:

I have a shiny dashboard application and my dataset is around 600 MB in size. It swells by 100 MB every month. My data resides locally in My

Answer 1

This is a very interesting question and deserves more proper responses rather than comments. I would like to relate my experience and thoughts. I built a commercial R+shiny application with Shiny Server Pro, using database(s) and loads of other tricks.

Delayed UI loading time
My app takes over 30s to load, i.e. to give control back to the user.

The issue

Shiny is a single page application. Therefore a complex app, with loads of tabs, data loaded to populate some of the menus & selectors etc. is affected and this starts from the initial loading time.

UI possible mitigations

• Use dynamic UI components (wisely) to add complexity after start up. For example a particular menu may start very simply with few elements, then add more elements at a later stage.
• Joe Cheng proposed insertUI and removeUI when my app was almost finished, so I didn't get around to use them, but they also could contribute to a simpler page for start up.

Use of database

My app used MonetDB and later PostgreSQL. The performance of MonetDB was good, but I had a multiple user conflict (complex issue that I cannot detail here) and this forced me to move to PostgreSQL as an alternative. PostgreSQL was fine, but it took a dramatic time to start due to the cache warming up issue. The design required to load at start up loads of data into the DB: bad design.

RDBMS delays possible mitigations

I think I tried most tricks with varying success.

• Limit RDBMS usage. As I decided from the start to use data.table to speed up data manipulations without been constraints by copying, I was also using fread for any type of csv reading. At the time fwrite (still from data.table) wasn't even on the horizon, otherwise it would merit serious considerations.
• App re-design. the app architecture has a lot to do with the degree of intensity that RDBMS are used. I'm convinced that time can be saved by a design that could take into account R+shiny (mainly R) limitations.
• Now MonetDB has R functions embedded into the code, so it should be even faster than before. It certainly deserves a good look. On the other hand the multi-user features should be thoroughly tested: most of R database code does not take into account to be used in a multi-user environment as offered by shiny. Maybe RStudio should be doing something more about this. Honestly they have already started, with the experimental introduction of connection pools and that is great.

Excessive use of reactivity

I think it is great to play with an advanced framework like shiny, and reactivity is a lot of fun to learn. On the other hand over a wide and complex application things can easily get out of hand.

Excessive reactivity possible mitigations

• Debugging each function gives a precise idea of how many time a particular shiny function is called, and any reactive function is called usually more than once. Of course all this burns cpu time, and needs at least to keep under control.
• Constructs like observeEvent now have parameters like ignoreInit: a wise use to these parameters can save at least a void cycle at initialisation time.

In my experience we have only scratched the surface of what it is possible to do with shiny. On the other hand there is a limit due to the single process nature of R. With Shiny Server Pro it is possible to envisage to use load balancers and spread multiple users across different servers. On the other hand to get into these territories we would need some kind of messaging system across the various instances. Already know I see the need for that in complex Shiny Server Pro applications (e.g. when there is the need to manage different classes of users, but at the same time to communicate between them). But this is out of scope to this SO question.