It all depends on what you are building and how you are building it. If you app is pretty straight forward rails app and you are not doing anything special on the system level (lets say working with transcribing video in a way that heroku add-ons would support it) than heroku is probably one of the fastest way to get started an move along. If you are are building something on the system level where you app needs to have access to anything on the system you probably better off going with linode or AWS route, It also depends alot about the level of control/responsibility your are looking for.
THere is also survey being done right now about this very topic, they havent published the 2012 results but 2009 are available:
http://rails-hosting.com/Results/SurveySummary.html