Some types of casting are so safe and efficient as to often not even be considered casting at all.
If you cast from a derived type to a base type, this is generally quite cheap (often - depending on language, implementation and other factors - it is zero-cost) and is safe.
If you cast from a simple type like an int to a wider type like a long int, then again it is often quite cheap (generally not much more expensive than assigning the same type as that cast to) and again is safe.
Other types are more fraught and/or more expensive. In most languages casting from a base type to a derived type is either cheap but has a high risk of severe error (in C++ if you static_cast from base to derived it will be cheap, but if the underlying value is not of the derived type the behaviour is undefined and can be very strange) or relatively expensive and with a risk of raising an exception (dynamic_cast in C++, explicit base-to-derived cast in C#, and so on). Boxing in Java and C# is another example of this, and an even greater expense (considering that they are changing more than just how the underlying values are treated).
Other types of cast can lose information (a long integer type to a short integer type).
These cases of risk (whether of exception or a more serious error) and of expense are all reasons to avoid casting.
A more conceptual, but perhaps more important, reason is that each case of casting is a case where your ability to reason about the correctness of your code is stymied: Each case is another place where something can go wrong, and the ways in which it can go wrong add to the complexity of deducing whether the system as a whole will go wrong. Even if the cast is provably safe each time, proving this is an extra part of the reasoning.
Finally, the heavy use of casts can indicate a failure to consider the object model well either in creating it, using it, or both: Casting back and forth between the same few types frequently is almost always a failure to consider the relationships between the types used. Here it's not so much that casts are bad, as they are a sign of something bad.