If there is a Chromecast around and:
After searching the Chromium source code, I figured out how to do it:
video::-internal-media-controls-overlay-cast-button { display: none; }
Example: http://jsfiddle.net/f1quhd2L/8/