This is more of a HTTP spec question than a Sitecore question.
I have a 404 error page set up in Sitecore:
As already mentioned correct http response headers are vital, not only for SEO but many other use cases.
There's a Sitecore Shared Source module called "Sitecore Error Manager" which makes your Sitecore installation respond with the correct 404 header and comes with a couple of other features like configuring any other error response, even for the Sitecore media library (i.e. pdf files), multilanguage support etc.
Sitecore Marketplace: http://marketplace.sitecore.net/en/Modules/Sitecore_Error_Manager.aspx
Github: https://github.com/unic/SitecoreErrorManager
As well as what Ruud has suggested, there was a blog post by Kern Herskind that uses a Sitecore/IIS approach without having to write any additional code.
That is a problem indeed. Because google will now think the 404.aspx
page does not exist and will think the non-existing item is temporarily redirected.
I came up with a solution for this and blogged about it here.
What it does basically is determine that the item is non-existing (through a pipeline processor), then retrieve the contents of the 404 page by requesting it using a WebRequest and return that contents to the browser with a 404 status code.