I think it's best to just test the public interface of an object. From the point of view of the outside world, only the behavior of the public interface matters and this is what your unit tests should be directed towards.
Once you have some solid unit tests written for an object you do not want to have to go back and change those tests just because the implementation behind the interface changed. In this situation, you've ruined the consistency of your unit testing.
