I\'m tearing my hair out over this one, so bear with me (it\'s a long post).
I had a similar problem sort of with WCF and using ASP.NET compatibility as I integrate MVC and WCF in my solution. What I would do is throw a WebFaultException then check the Status of the response at the receiving end (either java or other .NET client). Your custom error could then throw that if the WebOperationContext.Current is not null. You are probably aware of this already but just thought I would throw it out there.
throw new WebFaultException(HttpStatusCode.BadRequest);