Have you considered using a BinaryFormatter instead of the BinaryWriter?
Advantages
- You can pass objects (i.e. anything), so it solves your casting problem.
- Automatic type management (actually writes type headers to the stream).
- Supports complex reference types as well.
Disadvantages
Uses Serialization internally, therefore:
- Probably slower.
- Byte stream gets larger (because of the type headers).
- You don't have control over the byte format, therefore not an option in interop scenarios.
- Potential version issues (compatibility between different assembly versions of the serialized type).
- Requires the serialization code access permission (relevant in partial trust scenarios).