Is there any difference how the css3 effects (for example animation) are rendered if the doctype is defined as
(html5)
or
There is almost no difference* between those two doctypes in terms of CSS rendering.
The choice of doctype only affects validation, and which "browser mode" is used out of "Quirks Mode", "Standards Mode" ("no quirks mode"), "Almost Standards Mode" ("limited quirks mode").
See: http://hsivonen.iki.fi/doctype/
So, XHTML 1.0 Transitional
gives "Almost Standards Mode", whereas <!DOCTYPE html>
gives "Standards Mode".
*One difference is a minor and easily fixable adjustment concerning tables. Another is the treatment of heading elements nested inside certain HTML5 elements.
The doctype shouldn't affect this but there are libraries to help you determine what features are available to you in a particular client like Modernizr.
I think that HTML5 implies in itself that the CSS improves in readibility, nothing else.
Yes it does, and I can tell you that from personal experience.
HTML pages which consist of sliced tables that have no cellpadding and cellspacing, fail to work with a HTML5 doctype. I had to use HTML4 transitional.
HTML pages that use grid layout systems can sometimes work better with a HTML5 doctype. I find that the Tiny Fluid Grid does not work with a HTML4 transitional doctype.