This is more of a HTTP spec question than a Sitecore question.
I have a 404 error page set up in Sitecore:
As already mentioned correct http response headers are vital, not only for SEO but many other use cases.
There's a Sitecore Shared Source module called "Sitecore Error Manager" which makes your Sitecore installation respond with the correct 404 header and comes with a couple of other features like configuring any other error response, even for the Sitecore media library (i.e. pdf files), multilanguage support etc.
Sitecore Marketplace: http://marketplace.sitecore.net/en/Modules/Sitecore_Error_Manager.aspx
Github: https://github.com/unic/SitecoreErrorManager