Availability is nice, but even cheap solutions are usually sufficient.
Thing I've wanted most since I started having sites hosted: Shell access.
It is a PAIN to do everything through FTP and web interfaces.
Oh, and if your app has PHP/MySQL/Apache/IIS/.Net/JSP/etc version requirements, you'll want to check those first.