You could look at the ExecutionLog table in the ReportServer database. This will give you the break-up of times like data retrieval, processing, rendering etc. This could get you going in the right direction.
For example, in my previous project rendering reports in MS Excel was taking a lot of time - we switched some reports to be exported by default to CSV format and that helped. Similarly, we also identified bottleneck in the report processing and added more resources to the machine. This helped too.